Thread

Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

310 points · 140 comments · leonickson

jbird99 · 1785815708 · focus · HN ↗
At what, 10 tokens per hour? These disk swapping methods all have the same drawbacks - kill your drive early, and slow as hell.
1. kennywinker · 1785816307 · focus · HN ↗
  It says very prominently in the post: 4.5-5t/s for 80b on an M5
2. wat10000 · 1785816314 · focus · HN ↗
  Isn’t it only writes that kill drives?
  1. Alpha3031 · 1785817932 · focus · HN ↗
    Yes for NAND, and I suppose nobody is using mechanical hard drives for this.
    wat10000 · 1785865947 · focus · HN ↗
    I'd like to see someone try it, just to see how incredibly slow and noisy it is.
  2. sudo_cowsay · 1785818144 · focus · HN ↗
    Yeah, that's why most of these comments seem weird to me.
  3. petu · 1785822548 · focus · HN ↗
    There's read disturb on SSDs, enough reads will eventually force controller to rewrite the cell and it's neighbours.
    Practically if you're not streaming weights 24/7 from a full SSD, then it shouldn't be a problem.
    zozbot234 · 1785830681 · focus · HN ↗
    Read disturb ought to be quite rare, especially on a fresh drive that was written only once or a handful of times (WORM-like usage). Practically, it's not likely to be an issue even with very heavy read workloads.
    petu · 1785840011 · focus · HN ↗
    Hm, I asked 5.6 Sol to calculate/research and it came up with way less optimistic lifespan than Gemini (that I've asked last year).
    It claims that each individual page read induces read disturb across whole block. And references <a href="https://arxiv.org/pdf/2501.02517" rel="nofollow">https://arxiv.org/pdf/2501.02517 that tested Samsung 3D TLC and found ~518K sequential page reads in a block to be ECC threshold (although it's unclear how they got 518K number -- e.g. is it single worst chip they've tried? authors brings up 160 chip sample size later on).
    With 7704 pages in a block that's only ~70 sequential block reads till data is lost and to retain data controller would have to refresh block fair bit earlier.. basically it gives modern 3D TLC SSD lifespan measured in months (1TB drive 24/7 sequential reads at 5GB/s).
3. throwawayffffas · 1785833611 · focus · HN ↗
  Am I the only one that has no flash lifetime anxiety? I still have drives from more than a decade ago that keep on chugging fine.
  I remember the time spinning rust was the only option and reliable they weren't. In 30 years of computing I have had more than ten hdds and zero ssds die.
  1. a96 · 1785964653 · focus · HN ↗
    No, flash is very durable. SSD's do die and they generally just go completely dead with no warning or any way to try any recovery. I've had a few go so far and I have a lot of old SSD's go in and out of machines.
    No normal use should wear out a drive in any sensible time in reasonable use and even in unreasonable use they seem to last almost indefinitely. There's more likely to be some other kind of component death before that.
    Of course, in staged lab test, it's probably possible to burn one out. I've seen projects do that on memory cards and various *ROM chips but don't recall seeing someone kill SSDs. That would get costly. But I'm almost sure a web search would turn out someone doing that.
    I also used to be sure that 90's SCSI HDDs would never really stop running. Just the machines using them became too much work and no utility to keep going. I remember only one of mine that wouldn't start after some years in the storage, but I managed to hammer it back into shape.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.