Thread

1. Luker88 · 1791124296 · focus · HN ↗

   Currently I am running llama-cpp with `Qwen3.8-Flash-Next-UD-IQ3_XXS` on an old ryzen 8845HS with 96G of ram (and no dedicated graphics card) at 7tk/s and ~60tk/s filling, max ~120K context window.

   Surprisingly useful as long as you can leave it running a couple of hours at the very least.

   While huge models will still be better I think the general availability of RAM might be the downfall of AI companies.

   1. londons_explore · 1791125031 · focus · HN ↗

      Remember that a hosted AI company only needs ~1000 bytes per context token per user (ie. 100mb per user for typical coding - VRAM during inference, and moved to regular RAM or SSD whilst running a tool call)

      Everything else (weights) are shared amongst tens of thousands of users currently doing inference in that cluster, so even if there are terabytes of weights for the model, they aren't much on a per-user basis.

      1. eurekin · 1791125492 · focus · HN ↗

         > ~1000 bytes per context token per user

         Where's that figure comning from? Last time I checked (could be the 3.6 Qwen 27b) single token needed 32kb

         1. jburgess777 · 1791126567 · focus · HN ↗

            The smallest I have seen is DeepSeek 4.1 flash at 890 bytes per token.
   2. bitexploder · 1791131126 · focus · HN ↗

      Highly recommend the RCO-GSQ quant by ITSA btw. At IQ3_XSS it is within one point of the fully unquantized model.

      1. Luker88 · 1791133057 · focus · HN ↗

         will try, thank you for the pointer!

Unofficial Hacker News client; not affiliated with Y Combinator.