Thread

1. sneak · 1790665787 · focus · HN ↗

   Seriously though, what are the low cost chips that can usefully run LLMs? Is a Mac Mini the lowest we can go? Are there iGPUs on mini-itx that can do it, or are there dedicated AI chips that one could turn into a pi HAT?

   1. chorylee · 1790686952 · focus · HN ↗

      An Orange Pi 5 Max does this job for real. It's an RK3588 board — $75 for 4GB, $95 for 8GB on AliExpress. A community test got Qwen2.5-0.5B at about 12 tok/s on the CPU via llama.cpp. The chip also has a 6 INT8 TOPS NPU if you'd rather go the RKNN route. Won't beat a Mac Mini, but it's an actual computer for under a hundred bucks.

      1. cameron_b · 1790694757 · focus · HN ↗

         I regret to inform you that prices have long departed the lower atmosphere. Even on AliExpress, you're looking at a few hundred bucks for those. Still less than a Mac Mini, but less less.

Unofficial Hacker News client; not affiliated with Y Combinator.