Zos_Kia

I don’t know about other STTs but if you’re using whisper you can “prompt” it for consistent spelling. If you put “todo” in the prompt it should always spell it like that.

Have you tried using a vector DB with an embedder ? It may give decent performance without the need for a full blown LLM

I honestly wouldn’t recommend it if you don’t have a minimum of security knowledge. The moment your home server pops up with a domain name it will get scanned by shady actors and possibly exploited.

I use exclusively sshfs, including in my lan, is there some downside to it?

Same, I’ve been looking for something like that for quite some time

I am looking for a solution for a ~1TB collection, and the Glacier Deep Archive storage tier is barely above 1$/m for the lot. You may want to look into it ! If I remember correctly, the retrieval (if you one day need to get your data back) was around 20$ to get the data in a few hours, or 2$ to get it in a couple days.

If I understand these things correctly, the context window only affects how much text the model can “keep in mind” at any one time. It should not affect task performance outside of this factor.

Yeh, i did some looking up in the meantime and indeed you’re gonna have a context size issue. That’s why it’s only summarizing the last few thousand characters of the text, that’s the size of its attention.

There are some models fine-tuned to 8K tokens context window, some even to 16K like this Mistral brew. If you have a GPU with 8G of VRAM you should be able to run it, using one of the quantized versions (Q4 or Q5 should be fine). Summarizing should still be reasonably good.

If 16k isn’t enough for you then that’s probably not something you can perform locally. However you can still run a larger model privately in the cloud. Hugging face for example allows you to rent GPUs by the minute and run inference on them, it should just net you a few dollars. As far as i know this approach should still be compatible with Open WebUI.

There are not that many use cases where fine tuning a local model will yield significantly better task performance.

My advice would be to choose a model with a large context window and just throw in the prompt the whole text you want summarized (which is basically what a rag would do anyway).