Thread

Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

310 points · 140 comments · leonickson

dghlsakjg · 1785816328 · focus · HN ↗
I know everyone wants to crap all over these setups that are impractical, but this is how progress happens.
People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc.
Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
1. croes · 1785835579 · focus · HN ↗
  So every script kiddie gets its Mythos to hack sides and scammers don’t need AI services anymore.
  That time won’t be as much fun as you think
  1. cromka · 1785835817 · focus · HN ↗
    By that time the same Mythos class run on specialized hardware 5 years earlier will have fixed the same bugs that the Mythos class on consumer devices could have exploited.
    We'll be fine.
    bestham · 1785842488 · focus · HN ↗
    No, CVEs are on average still actively being exploited for far longer than 5 years. The oldest CVEs on CISA’s KEV (Known Exploited Vulnerabilities) catalog are from early to mid 2000s.
    cromka · 1785846323 · focus · HN ↗
    That's status right now, and that's because we just got models than can speed up the mitigation or find new CVEs. Plenty of reports of how many of them get patched now, most recently with Chrome.

Unofficial Hacker News client; not affiliated with Y Combinator.