I’m sick and tired of the Capitalist religion and all their fanatic believers. The Western right-wing population are the most propagandized and harmful people on Earth.

  • 0 Posts
  • 9 Comments
Joined 3 years ago
Cake day: June 8th, 2023
  • Sims@lemmy.mltoSelfhosted@lemmy.world•What budget friendly GPU for local AI workloads should I aim for?English
    2·
    11 months ago

    Compute prices are not going down. If you want a budget friendly compute, then you may need to look for NPU’s. They are slower, and is unfortunately also getting more expensive pr TOPS. Electricity prices in the West are going up as a low output gets bought up by datacenters instead. The newest GPU’s are more efficient in re. to power pr Top, but they are not budget friendly.

    I have not seen a recent comparison of GPU vs Npu; TOPS vs price vs power consumption, but @ aliexpress, I saw a small 40 TOPS Npu as a nvme stick with 16gb ram that draws 10 watts, or so (search for ‘ai accelerator nvme’). This little thing can scan files 24/7 and your i5 can help out in peaks. Afaik you can run a distributed model that also runs in, and uses your i5 memory/compute, so if you max out i5 memory, perhaps the combined compute is enough for a larger model ? Maybe a few Npu sticks can work together on the same model ?

    Alternatively, you could look for the next version of a Huawei gpu card (first had baby kinks afaik), or one of the other upcoming producers from China. They’ll come faster and faster, but are earmarked for local consumption first.

    Another suggestion is to buy one of the old P40/80 (from the ‘pascal’ chipset, I think) (or was it k40/80 ??). They should still support a descent range of modern quantization needs and often have 24gb ram. A refurbished miner card cost around 50-70$+ - cant remember exactly tho.

    Lastly, you could try something different. If the i5, with enough memory, can run a large model slowly, you could add a dedicated KV cache, so most of the tokens won’t need to be recalculated. Memory and bandwidth are the most important here, but any old server could be upgraded to be a dedicated KV cache server (might need a network upgrade tho!).

    Anyway, Ideas are easy and cheap. If I were you, I would ask an AI for a little python app where you can add a product and it returns a graph where it compares with other products and show optimality over time given prices/power and Tops - Gpu vs Npu. Good hunting…

  • Sims@lemmy.mltoSelfhosted@lemmy.world•Is there any good decentralized cloud storage for personal backups as a self-hoster?English
    2·
    1 year ago

    I don’t know, but allow me a soft rant about the ‘distributed’ part;

    Couldn’t selfhosters try to ‘organize’ and share these burdens ? Why pay for external cloud backup, or anything, when selfhosters can just help each other storing parts of others backup. Then everyone have an automatic back-up.

    The tools seem to be there, but Its like there are all these super-skilled infra-structure selfhosters that know everything about self-hosting solutions, but they lack the self-organizing ability to sollve these typical - and a bit trivial, lets be honest - problems in a full p2p style. The result looks to be that all self-hostings solution above the threshold of an average individual selhoster, have to be done in the cloud, and everyone is ‘siloed’ in their own mini data center.

    But, with existing tools, AI and a little imagination, it shouldn’t be too hard to ‘organize’ a little (though here) design a self-hosting p2p backup solution from existing tools. …or a solution for most of the other cloud services we still rely on…

    But maybe its something else ? …to me, it just seems unnecessary for a high expertise self-hoster community that - when combined - are an absolute gargantuan cloud-service infrastructure …to still have such basic capacity issues (no offense meant to op, or anyone!), and still have so high reliance on cloud services. Seems odd to me…

  • Sims@lemmy.mltoSelfhosted@lemmy.world•Would you use a self-hosted, AI-powered search engine for your favorite sites?English
    34·
    1 year ago

    Absolutely. I’m not as AI-phobic, and I absolutely need both an information gathering agent that searches, follows rss/channels, explores/researches topics, summarizes swats of daily events, and I need a filter between me and corporate internet. Let the AI discard obvious slop, Ads/other propaganda and general informational noise.

    However, you should think about how to share search-results so all our agents don’t floor small services/commoners. We really need to get more information out of corporate silo’s, and into public search/knowledge systems. Perhaps you can integrate some distributed search/knowledge properties ? (now that you have AI to help build it)