- YouTube
youtu.be
Enjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.

*Western AI companies are terrified

Anyway, what do you all think of the news?

Video is 8 minutes and 30 seconds long if you’re interested.

Is it true that the DeepSeek models and that other AI model (I forgot the name already lol) can be run locally? How do they compete with Claude and OpenAI?

Also, what about that other AI thing, OpenClaw?

I am interested in AI but still very much skeptical.

I have only used Duck AI sometimes but try to avoid it.

  • Faux@lemmygrad.ml
    8·
    2 months ago

    DeepSeek V4 flash is feasiblish to run it personally. A minipc costing $3000 can do a working quantization, two such computers can run what is in practice a full model. You could even run a (weak but working) quantization of GLM5.2 in this configuration.

    What you get running models this way is control over what exactly runs (you never know how aggresively quantized are models running with API, for example) and possibility of running anything that fits (e.g. models with refusals removed with methods like heretic).

    What you lose is that you need to buy special hardware for thousands of dollars (which might or might not age well) and loading prompt (including the history of conversation) can be really slow compared to what bigger machines that host models for many users can achieve. Loading past context of 100,000s tokens takes actual minutes while using the same model through API will feel instant.

    It shows how important is for people to own the datacenters. Running model on bigger hardware for multiple people is much more efficient than running it on personal hardware.

    • darkernations@lemmygrad.ml
      6·
      2 months ago

      A minipc costing $3000 can do a working quantization, two such computers can run what is in practice a full model…

      Would you able to please spec such a PC?

    • Faux@lemmygrad.ml
      5·
      2 months ago

      About how strong these models are compared to claude - DeepSeek flash is much better than Haiku, GLM 5.2 is comparable at level of Opus, definitely better than Sonnet. In a quantization you could run with 256GB of VRAM, it would still be better than Sonnet I think.

      You can run these models with API and save a lot of money compared to Claude too if you swap: Haiku -> DeepSeek Flash, Sonnet -> DeepSeek Pro, Opus -> GLM 5.2, Fable -> Kimi K3

    • Rylo@lemmygrad.ml
      4·
      2 months ago

      It shows how important is for people to own the datacenters. Running model on bigger hardware for multiple people is much more efficient than running it on personal hardware.

      Which also is why the AI companies are panicking a little bit. They don’t necessarily target private consumer anyways, but are trying to expand into the business-licensing space. Thats why they are competing so hard to create the most complete ecosystem of tools, i.e. coding, legal work, picture+movie, defense.

      But no reasonably lead company is going to waste $9999999 billions on tokens for ever more expensive models when you can get similar results by expanding your serverhall with a few m^2.

    • Makan@lemmygrad.mlOPEnglish
      2·
      2 months ago

      Interesting…

      Can you tweak or change the AI completely?

      Or even somehow notifies revamp it into your own AI completely?

      Maybe build off of it somehow and improve it maybe?

      • Faux@lemmygrad.ml
        4·
        2 months ago

        If you mean using different models, yes. If you mean finetuning model weights… No, finetuning LLMs is hard in general and Strix Halo (device that I had in mind that costs 3K$ for 128GB of VRAM) is also kinda bad for it now due to having an AMD (and not really strong when it comes to compute power) GPU.

        In practice, people who try finetuning on top of newest models usually cripple them. There are plenty of finetunes of Qwen 3.6 on huggingface trying to distill stronger models and they are just bad.

        Heretic, the method for removing refusals I mentioned, also requires more VRAM to run it than it’s required for inference.