‹ BackHN Continuity

Thread

Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

310 points · 140 comments · leonickson

  1. AHASIC · · focus · HN ↗
    I read a comment on here a few months back I wanna restate. Basically, there is a good chance that Apple is betting that the LLMs in the future will be so efficient that those that consumers will use everyday will be easily computed by the iPhone or even bigger ones on Macs. Honestly makes the most sense that we are heading that way in a few years latest.
    1. Mistletoe · · focus · HN ↗
      What hardware advances would we need to see for that to happen? It feels like everything in that arena has kind of plateaued.
      1. CircuitSeuss · · focus · HN ↗
        A lot of this will come from co-optimizing hardware and low level machine code for this specific use case… something apple is coincidently very good at.

        Apple has worked very hard to make unified memory a feasible approach, and the benefits of that are pretty clear in apple silicon- that efficiency not only results in power and therefore thermal gains, but also in a significantly faster full loop per process: or a faster time to token. This is why even their single core mobile chips in the budget line Neo out perform PC processors with several times more threads and RAM[1]. Turns out, unified memory lets you have a whole lot more control over things like RAM bussing and core use for specific workflows.

        Speculatively, a unified memory approach could also allow you to more easily integrate things like ReRAM to solve the current memory swapping bottleneck.

        Let’s say a friend of mine works hardware at apple and works on exactly this… on device processing is the future I’m betting on.

        [1] <a href="https:&#x2F;&#x2F;youtu.be&#x2F;x26A28DoT-w?t=605" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;x26A28DoT-w?t=605

        1. jsjohnst · · focus · HN ↗
          &gt; This is why even their single core mobile chips in the budget line Neo

          That’s a six core processor. It’s an A18 Pro in the Neo, same chip as on the Iphone 16 Pro

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.