Thread

Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

310 points · 140 comments · leonickson

CyLith · 1785818827 · focus · HN ↗
I know relatively little about the workings of LLMs, but I keep seeing projects like this that run massive MoE models using very modest amounts of RAM, perhaps excessively so. I wonder, is there a way to make the RAM usage tunable? I have a Macbook with 32 GB of RAM, and it'd be great if I could run the same model but take advantage of the additional RAM to make it run faster.
1. fodkodrasz · 1785828187 · focus · HN ↗
  > I wonder, is there a way to make the RAM usage tunable?
  In LM Studio I can tune it by selecting different quantation of the model, by selecting how many layers of the neural net to be loaded to GPU (rest stays in main mem, evaluated by the CPU), and by adjusting context window.

Unofficial Hacker News client; not affiliated with Y Combinator.