‹ BackHN Continuity

Thread

Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

310 points · 140 comments · leonickson

  1. CyLith · · focus · HN ↗
    I know relatively little about the workings of LLMs, but I keep seeing projects like this that run massive MoE models using very modest amounts of RAM, perhaps excessively so. I wonder, is there a way to make the RAM usage tunable? I have a Macbook with 32 GB of RAM, and it'd be great if I could run the same model but take advantage of the additional RAM to make it run faster.
    1. zamadatix · · focus · HN ↗
      That's about the turning point for just using typical quants for me. Larger still and you can just do the full model. Smaller to this degree and you need all sorts of extra tricks to get anything.
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.