Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
Thread
Unofficial Hacker News client; not affiliated with Y Combinator.
Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
Unofficial Hacker News client; not affiliated with Y Combinator.
dghlsakjg · · focus · HN ↗
People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc.
Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
ksec · · focus · HN ↗
I am 100% sure this won't happen in 10 years time. At least not on a $200 SSD. But I wouldn't be surprised if it ran on $1K to $2K HBF SSD. It will still be better than a $50K Rack.
spacebacon · · focus · HN ↗
[dead]