‹ BackHN Continuity

Thread

Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

310 points · 140 comments · leonickson

  1. dghlsakjg · · focus · HN ↗
    I know everyone wants to crap all over these setups that are impractical, but this is how progress happens.

    People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc.

    Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

    1. myrmidon · · focus · HN ↗
      Agree with this. As soon as things get in range for motivated amateurs, progress skyrockets.

      Has also been the case for things like chess computing; a lot of the progress we made over the last decades there (even before involving neural networks!) happened thanks to software improvements because the problem got so accessible, not just faster hardware.

      I expect similar trends with AI; I'd expect to get decent, human comparable capability with <200GB/s of memory bandwidth and under 60GB of RAM long term (SSDs with very high read bandwidth looks also promising, but we'll see).

      I hope that in a decade or two, training will also be somewhat feasible for "pro-sumers".

      1. Bayart · · focus · HN ↗
        > As soon as things get in range for motivated amateurs, progress skyrockets.

        That's how the AI boom came around to start with, before CUDA it was an academic topic.

      2. mannanj · · focus · HN ↗
        Is this where with in conjunction with Jevon's paradox, we are AI usage to exponentially skyrocket?
      3. RachelF · · focus · HN ↗
        >As soon as things get in range for motivated amateurs, progress skyrockets.

        So true. The reverse is also true - when greedy companies overprice their initial release so that it is out of range of the enthusiastic hobbyist they stall progress and adoption.

        This is true for hardware (eg failed Intel Optane, Knights Bridge) and software that does not have a cheap or free basic plan.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.