Thread

Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

310 points · 140 comments · leonickson

jbird99 · 1785815708 · focus · HN ↗
At what, 10 tokens per hour? These disk swapping methods all have the same drawbacks - kill your drive early, and slow as hell.
1. wat10000 · 1785816314 · focus · HN ↗
  Isn’t it only writes that kill drives?
  1. Alpha3031 · 1785817932 · focus · HN ↗
    Yes for NAND, and I suppose nobody is using mechanical hard drives for this.
    wat10000 · 1785865947 · focus · HN ↗
    I'd like to see someone try it, just to see how incredibly slow and noisy it is.
  2. sudo_cowsay · 1785818144 · focus · HN ↗
    Yeah, that's why most of these comments seem weird to me.
  3. petu · 1785822548 · focus · HN ↗
    There's read disturb on SSDs, enough reads will eventually force controller to rewrite the cell and it's neighbours.
    Practically if you're not streaming weights 24/7 from a full SSD, then it shouldn't be a problem.
    zozbot234 · 1785830681 · focus · HN ↗
    Read disturb ought to be quite rare, especially on a fresh drive that was written only once or a handful of times (WORM-like usage). Practically, it's not likely to be an issue even with very heavy read workloads.
    petu · 1785840011 · focus · HN ↗
    Hm, I asked 5.6 Sol to calculate/research and it came up with way less optimistic lifespan than Gemini (that I've asked last year).
    It claims that each individual page read induces read disturb across whole block. And references <a href="https://arxiv.org/pdf/2501.02517" rel="nofollow">https://arxiv.org/pdf/2501.02517 that tested Samsung 3D TLC and found ~518K sequential page reads in a block to be ECC threshold (although it's unclear how they got 518K number -- e.g. is it single worst chip they've tried? authors brings up 160 chip sample size later on).
    With 7704 pages in a block that's only ~70 sequential block reads till data is lost and to retain data controller would have to refresh block fair bit earlier.. basically it gives modern 3D TLC SSD lifespan measured in months (1TB drive 24/7 sequential reads at 5GB/s).

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.