Thread

Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

310 points · 140 comments · leonickson

dghlsakjg · 1785816328 · focus · HN ↗
I know everyone wants to crap all over these setups that are impractical, but this is how progress happens.
People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc.
Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
1. arjie · 1785817257 · focus · HN ↗
  Haha 1T on $50k might be a bit hopeful, mate, even at FP8. But I too am hopeful.
  1. hedora · 1785817651 · focus · HN ↗
    AMD already demonstrated 1T on strix halo clusters. << $10K at original MSRP.
    arjie · 1785873952 · focus · HN ↗
    Problem is prefill on these, right? Initial prompt processing takes forever? I suppose you’re right. Cost is not a thing on its own. It’s a performance-cost frontier and one can do CPU inference in the worst case.
  2. apimade · 1785820168 · focus · HN ↗
    8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS.
    1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS.
    5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today. Theoretically around 104.8 TFLOPS.
    In the same timeframe mobile processor CPU's went from 0.001 TFLOPS, to today's Apple's A19 Pro chip which delivers 2.074 TFLOPS.
    That's _without_ getting into ASIC's, or purpose-built hardware like Taalas's model on silicon HC1, or generic AI dies like what they're planning with HC2 or Cerebras, which will massively compress the timeline.
    cududa · 1785824681 · focus · HN ↗
    Just a note that I think the direction most people are paying attention to is memory bandwidth; thats the real bottleneck and “number go up” but also constraint people are designing around
    xbmcuser · 1785825410 · focus · HN ↗
    We will see such power and price now only when AI market crashes or China reaches node parity and goes after market share as currently the way they are buying out most of the latest node production the consumer prices will only be palatable to the very rich or we will need to be happy with older slower nodes
    formerly_proven · 1785825576 · focus · HN ↗
    Except the 499$ of a 1080 GTX inflation-adjusted only buys you a 5070 or 5070 Ti even by MSRP.
    zmmmmm · 1785825969 · focus · HN ↗
    Sadly while the FLOPS are increasing nicely, total graphics memory is stalled in consumer cards by comparison.
    jack_pp · 1785826217 · focus · HN ↗
    Isn't there such a thing as low hanging fruit?
    Aren't we already approaching theoretical physical limits? We're at 2nm
    kaashif · 1785828861 · focus · HN ↗
    (1) Yes.
    (2) Are you saying that you think we're at the limits of computing in general, or that specific technology?
    We know, for example, that a human brain level intelligence is possible to run on a human brain. We are nowhere near that. And actually that's not even a physical limit necessarily.
    But that is...not a low hanging fruit.
    fragmede · 1785829045 · focus · HN ↗
    > We are nowhere near that.
    Nowhere?
    darkwater · 1785831720 · focus · HN ↗
    Leaving aside the discussion on LLMs intelligence vs human intelligence, on a purely energy consumption level we are definitely and without any possible questioning nowhere near that indeed.
    cvak · 1785830224 · focus · HN ↗
    we are not at 2nm, we just call it that.
    root_axis · 1785826900 · focus · HN ↗
    Ok, now do memory capacity and bandwidth - the things that actually constraint local LLMs.
    apimade · 1785830928 · focus · HN ↗
    8800 GTX in 2006: 768 MB of GDDR3, with 86.4 GB/s of theoretical memory bandwidth.
    GTX 1080 in 2016: 8 GB of GDDR5X, with 320 GB/s.
    RTX 5090 in 2026: 32 GB of GDDR7, with 1.792 TB/s.
    This is fun, what's next?!
    PCI 8.0 is breaking 1TB/s, GDDR7 is 1TB/s.
    With just the _current_ timeline, things are looking like they'll compress once we get over this initial lump.
    flaburgan · 1785828300 · focus · HN ↗
    Yeah but here you describing the opposite phenomenon. You're saying that the hardware is going to become cheaper and more powerful with the years, to the point a current State of the Art model from today will run on a normal consumer hardware in ten years. What people are trying to do now is the opposite, optimize the software as much as possible so that it does not need the best hardware but the normal one we currently have. As if we were trying to make a current AAA game to run smoothly on the 1080 GTX of your example.
    dtj1123 · 1785835753 · focus · HN ↗
    No, both of these things can happen in parallel.
    The suggestion is that a 1T model could be made to run on cheap consumer hardware of the future.
    foxrider · 1785831977 · focus · HN ↗
    Speaking of ASICs - how likely is it that as models get better we'll see someone baking a whole model directly into the silicon? It's like having l0 cache.
    SJC_Hacker · 1785833482 · focus · HN ↗
    You could do it but there would be no point, The only advantage over would be power consumption. And it would be quite expensive.
    At the rate models are improving, it would be obsolete in six months.
    HPsquared · 1785834798 · focus · HN ↗
    Power consumption and latency are very important on mobile
    naasking · 1785835752 · focus · HN ↗
    They're important everywhere of course, but especially on mobile. If AI researchers figure out how to offload knowledge and expertise from reasoning weights, then a core reasoning ASIC linked to the knowledge would totally rock.
    foxrider · 1785843013 · focus · HN ↗
    Yes, right now it would be obsolete in six months, but I also must add that this never stopped crypto miners from making new ASICs. However, with how useful Kimi is right now - at some point if someone makes a dedicated hardware board with "good enough" model for daily tasks - that would be a very sought after commodity.
    kaelwd · 1785835060 · focus · HN ↗
    Only 8B currently but it's been done: <a href="https://taalas.com/products/" rel="nofollow">https://taalas.com/products/
    apimade · 1785835955 · focus · HN ↗
    The only _public_ example we know.
    This is definitely being done with private models by HFT/quant firms, data processing agencies/orgs (large intelligence agencies, _every_ data analytics org, etc).
    Azantys · 1785839857 · focus · HN ↗
    Here you have shown yourself that progress slows down and doesnt speed up. 8.9/0.35 = ~25x more performance in 10 years from 2006 to 2016. 104.8/8.9 = ~12x more performance in 10 years from 2016 to 2026. Growth has dropped 50%.
    dghlsakjg · 1785850261 · focus · HN ↗
    That isn't deceleration, you've just chosen a very selective way to compare. If you use time as a denominator, which is kind of intrinsic when talking about rates of acceleration, you get a very different result. If you graphed .3, 8.9, and 104.4 on the y axis, with years on the x axis, it would be pretty clear that there was in increase in the rate of progress.
    We went from adding 8 teraflops in a decade, to adding almost 100 the next decade. If we add "only" 400 more teraflops in the next decade the graph will make that initial growth look flat in comparison, even though your math would show that we are basically stalled out.
    It’s like claiming that a company that goes from making $1 to $1k to $100k to $1mm in a 4 year period has decelerating growth.
    ksec · 1785864224 · focus · HN ↗
    >It’s like claiming that a company that goes from making $1 to $1k to $100k to $1mm in a 4 year period has decelerating growth.
    Because it is decelerating growth. There is a reason why we use YoY percentage in annual and financial reporting.
  3. sh3rl0ck · 1785830886 · focus · HN ↗
    We'd bought 4 x $11K Mac Studios at my college and via exo, we had Kimi K2.5 at 30 TPS.
    Not too wild an idea!
    arjie · 1785873873 · focus · HN ↗
    30 tok/s generation? Nice! What was prefill by the way? Also, these are 256 GB RAM studios? Good timing on those!
2. gpt5 · 1785824544 · focus · HN ↗
  Most people are already used to rely on the internet on basically everything. At best, they download a tiny chunk of entertainment from it when they go on a plane, and as soon as they land they immediately abandon that offline chunk.
  In addition, LLMs, small or large, are highly parallelizable. This means that running on the same machine/GPUs many requests in parallel is significantly more efficient, and the sum of tok/s will far outweight any single user use case.
  Those two combined means that unless LLMs reach the state of 'good enough' (TBD), I fully expect the economics and people's interest to align with 99%+ of LLM usage would be in centralized servers. (not dismissing the fact that there are use cases for local LLMs, and that the progress matters).
  1. anon373839 · 1785825976 · focus · HN ↗
    I suspect the economics favor centralized servers, if you only look at the aggregated cost to serve X number of users' tokens. But we could say the same thing about a lot of the computation that iPhones do locally. They could have been much thinner clients, but instead they now have more compute power than desktops had when iPhones launched.
    gpt5 · 1785827087 · focus · HN ↗
    iPhone are in practice more reliant on central servers than ever before. Except for some games, if you take a random person's iPhone it becomes almost useless without internet connection. Which is exactly my point, it's not about compute, it's about people's expecations and priorities. The main driver of the need for higher performance chips on iPhones is its camera and smooth 120fps UI animations, not local processing for apps for almost all people.
    However, the compute situation just skew it even further when it comes to LLMs.
    anon373839 · 1785827772 · focus · HN ↗
    Phones are communication devices. So I would always expect connectivity to be central to their use. But that's a different issue than the "inefficiency" of edge computation and whether powerful edge devices can sell (they obviously can).
    xyzsparetimexyz · 1785830146 · focus · HN ↗
    A lot of the compute that iPhones do either has high bandwidth requirements (photo editing etc) or low latency requirements (games etc). LLMs are an interesting case where unless you're feeding in images, it's just text
    zozbot234 · 1785831394 · focus · HN ↗
    > I suspect the economics favor centralized servers, if you only look at the aggregated cost to serve X number of users' tokens.
    The economics of real-time, low-latency inference of very large near-SOTA models will heavily favor a centralized setup. But if you can afford to wait for your answer - be it a day, a week, or even more at the extreme low end (or if you just stick to leaner models for your relatively quick replies) the economics start to shift in a very clear way. A slow-going local inference setup relying on cheap SSD offload does not need the high power input of a datacenter rack, and the cooling load is outright trivial - even when working on many requests in parallel, which (in a SSD offload context) is what maximizes throughput even for local inference. These are serious problems for centralized inference that will probably limit the scale at which it can be applied.
    dghlsakjg · 1785980032 · focus · HN ↗
    iPhone, at this very moment, ships with an llm that third party developers can use. People are already using local llms.
  2. submeta · 1785829895 · focus · HN ↗
    I don’t understand the downvotes. It’s perfectly reasonable to disagree, but the argument is laid out clearly. Why not engage with it instead?
  3. mihaaly · 1785834857 · focus · HN ↗
    This is a nice personal point of view. And you don't care much about privacy or being exposed to the conduct of others completely, do you?
  4. hobofan · 1785835762 · focus · HN ↗
    > Those two combined means that unless LLMs reach the state of 'good enough'
    Once you are able to not just go on-premise, but on-device, there a boatload of other advantages than efficiency (network independence, privacy, etc.) that make it attractive.
    Tiny language models are already plenty "good enough" for a lot of on-device use-cases. E.g. there are a lot of things, that can be reformulated as ranking problems, where you can apply pairwise ranking, which even small language models are good at.
  5. dudefeliciano · 1785843822 · focus · HN ↗
    > Most people are already used to rely on the internet on basically everything
    FAANG has trained them well. "Normal" people also use chatGPT & co. to ask mundane questions that qwen27b could easily answer (random trivia, writing emails, correcting translations etc), why do these simple requests need to be processed on a B300 on server on the other side of the planet? There is no reason.
    And this being hackernews, I would expect people here to go against the grain in this regard...maybe even promote local llms to "normies".
3. marci · 1785828486 · focus · HN ↗
  Seems like what Apple's going for with afm3. Their latest model that will be embedded in macOS 27 is a quantized dense 20B that only select between 1 to 4B at inference, based on the prompt, not token by token. If only they could make a 100B or 400B dense that selects ~5 to 15B...
  1. josu · 1785831536 · focus · HN ↗
    I don't understand, if they are only using a subset of the tokens then it's a sparse model. What do you mean by dense?
    l33tman · 1785831970 · focus · HN ↗
    Could it be some sort of permanently routed MoE where they detect and switch for the whole prompt instead of token by token?
    marci · 1785859515 · focus · HN ↗
    Nothing to understand. Straight up hallucination. I could have sworn I read that they used a novel architecture where the model is dense but you could select specific layers or something at inference. reread the announcement: just said MoE. Corrected my brain's weights so thanks.
    <a href="https://machinelearning.apple.com/research/introducing-third-generation-of-apple-foundation-models" rel="nofollow">https://machinelearning.apple.com/research/introducing-third...
    josu · 1785912367 · focus · HN ↗
    Thanks for responding.
4. croes · 1785835579 · focus · HN ↗
  So every script kiddie gets its Mythos to hack sides and scammers don’t need AI services anymore.
  That time won’t be as much fun as you think
  1. cromka · 1785835817 · focus · HN ↗
    By that time the same Mythos class run on specialized hardware 5 years earlier will have fixed the same bugs that the Mythos class on consumer devices could have exploited.
    We'll be fine.
    bestham · 1785842488 · focus · HN ↗
    No, CVEs are on average still actively being exploited for far longer than 5 years. The oldest CVEs on CISA’s KEV (Known Exploited Vulnerabilities) catalog are from early to mid 2000s.
    cromka · 1785846323 · focus · HN ↗
    That's status right now, and that's because we just got models than can speed up the mitigation or find new CVEs. Plenty of reports of how many of them get patched now, most recently with Chrome.
  2. JKCalhoun · 1785849560 · focus · HN ↗
    If that's the natural outcome then the future you're describing is inevitable.
    Oh well, maybe the last one to leave the internet can turn the lights off.
    croes · 1785867023 · focus · HN ↗
    It’s not a natural outcome it’s a decision
5. myrmidon · 1785836525 · focus · HN ↗
  Agree with this. As soon as things get in range for motivated amateurs, progress skyrockets.
  Has also been the case for things like chess computing; a lot of the progress we made over the last decades there (even before involving neural networks!) happened thanks to software improvements because the problem got so accessible, not just faster hardware.
  I expect similar trends with AI; I'd expect to get decent, human comparable capability with <200GB/s of memory bandwidth and under 60GB of RAM long term (SSDs with very high read bandwidth looks also promising, but we'll see).
  I hope that in a decade or two, training will also be somewhat feasible for "pro-sumers".
  1. Bayart · 1785839129 · focus · HN ↗
    > As soon as things get in range for motivated amateurs, progress skyrockets.
    That's how the AI boom came around to start with, before CUDA it was an academic topic.
  2. mannanj · 1785848765 · focus · HN ↗
    Is this where with in conjunction with Jevon's paradox, we are AI usage to exponentially skyrocket?
  3. RachelF · 1785879121 · focus · HN ↗
    >As soon as things get in range for motivated amateurs, progress skyrockets.
    So true. The reverse is also true - when greedy companies overprice their initial release so that it is out of range of the enthusiastic hobbyist they stall progress and adoption.
    This is true for hardware (eg failed Intel Optane, Knights Bridge) and software that does not have a cheap or free basic plan.
6. pizza234 · 1785836695 · focus · HN ↗
  > but this is how progress happens.
  This is progress in the same way that a man climbing a tree is making progress toward reaching the moon.
  This project is essentially the MoE-of-the day, with some platform-related optimizations.
  > Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
  That won't happen. Projects like this just give the illusion that that will be possible.
  1. AlecSchueler · 1785840487 · focus · HN ↗
    > This is progress in the same way that a man climbing a tree is making progress toward reaching the moon.
    That is, an essential step?
  2. rhdunn · 1785840520 · focus · HN ↗
    There are several aspects to this:
    1. Better hardware optimized for matrix operations and other ML/NN specific workflows (NVIDIA hardware, Apple M4/M5 chips, Intel AI Boost, other NPU/TPU chips, etc.).
    2. Bigger RAM/VRAM/unified for loading models in memory -- with 3090/4090 you can run 22-24B models in VRAM; with 5090 you can run 30-32B models; with Apple hardware with 128GB unified RAM you can run 70-128B models.
    3. Better quantization (smaller, better value representations/lower errors/loss, optimized CUDA/etc. kernels) will lead to larger models being able to be run on the lower VRAM hardware at (with optimized kernels) decent tokens/s.
    4. Better optimizations in general taking advantage of the hardware (coupled with the hardware and driver advances) and the NN architectures (NOTE: I saw a huge performance boost a while back after upgrading my llama.cpp version).
    5. Better model architectures optimized/designed for running on lower-spec hardware, like Mixture-of-Experts which make it easier to run on a single GPU or distribute the weights more efficiently across multiple GPUs (i.e. you know that specific weights will be used together so you can keep them resident on a specific GPU and not have to swap the weights in/out of the various GPUs or transfer intermediate results of a single layer, only the result from that block/expert).
    5. Improved NVME or similar SSD technology which a) has a large amount of storage, and b) has access efficiency on par with RAM/VRAM, especially if/when coupled with efficient DMA access to the GPU/NPU.
  3. sipjca · 1785840520 · focus · HN ↗
    Unfortunately you are wrong. Time marches on and progress in silicon does too. You may not like the timeline, but it will happen.
    While not exactly an “SSD”, High Bandwidth Flash will make it to market and make big sparse MOE models even more accessible. Everyone knows bandwidth is the game right now and you think hardware vendors are going to sit on their heels and not take part of the market? Absolutely silly to discount progress both on the software and hardware sides.
  4. Hugsbox · 1785851351 · focus · HN ↗
    "It will never be possible to have a computer in your home. You will always have to buy computer time and access via dumb terminals. Technology will never again advance."
  5. trymas · 1785853570 · focus · HN ↗
    it’s equivalent take to laugh at first transformers 9 years ago, because they were shit and hardware requirements were immense.
    IMHO it’s a matter of time until we (consumers) will get the hardware (maybe coupled maybe even more novel techniques). Though I expect it will take another 10 years or more.
    mv4 · 1785864088 · focus · HN ↗
    The frontier labs will do their best to prevent this from happening. Their financial model won't work if people start running open-weight models on their local hardware.
    This is why banning Chinese open-weight AI models is a major policy debate in Washington. The labs can't survive log-term without subsidies, and a ban can act as a subsidy.
    trymas · 1785917462 · focus · HN ↗
    I concur, that this could be a possiblity and it definitely seems now.
    Though my bet would be, if USA will go ultra protectionist in this regard - in 10-20 years most world will run Chinese LLMs and hardware for this purpose.
  6. bluerooibos · 1785855798 · focus · HN ↗
    > That won't happen
    I don't understand how anyone with any degree of technological or scientific literacy or knowledge of advances in the past 100-200 years can say something won't happen in the future.
    I am absolutely certain it will - the only uncertainty is how long, but given the rate of progress, I'm guessing it won't be far away.
  7. mv4 · 1785864307 · focus · HN ↗
    Technically, this can happen. Politically, this won't be allowed to happen - just like digital media ownership never happened.
7. hagen8 · 1785839352 · focus · HN ↗
  There are certain physical limits. Calculations need to be done. Either less calculations are necessary for the intelligence, or u accept less intelligence. But there is a limit in what u can do with specific hardware.
8. Perz1val · 1785840445 · focus · HN ↗
  > wearing the hard drive
  Do reads wear out nvme? Iirc only writes do
  1. efficax · 1785842976 · focus · HN ↗
    only writes and honestly that’s a bit overblown they last a long time
9. jarek83 · 1785845042 · focus · HN ↗
  If 1t param model would run from SSDs those will stop costing only $200.
10. JKCalhoun · 1785849330 · focus · HN ↗
  Agree—and I want to see this functionality move to the larger platforms that everyone uses like LM Studio.
  1. ac29 · 1785852106 · focus · HN ↗
    llama.cpp has used mmap by default for years
11. ksec · 1785863947 · focus · HN ↗
  >when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
  I am 100% sure this won't happen in 10 years time. At least not on a $200 SSD. But I wouldn't be surprised if it ran on $1K to $2K HBF SSD. It will still be better than a $50K Rack.
  1. spacebacon · 1785865378 · focus · HN ↗
    [dead]

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.