Thread

Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

826 points · 365 comments · snehesht

a11r · 1791132704 · focus · HN ↗
I'm a little skeptical of going below 4-bit quants due to the potential for significant degradation in quality. I'm running 4-bit quants on an RTX Pro 6000 rented for approximately $1/hour and getting about 1.2 million tokens out and 40 million tokens in per hour with caching. The quality of 4-bit quant is good enough for difficult but well-scoped coding tasks. Here is the inference stack I am using: <a href="https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK" rel="nofollow">https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK
1. sudo_cowsay · 1791159572 · focus · HN ↗
  For the 1 dollar per hour, may I ask (without trying to incite things) why you would not just buy a ChatGPT Plus subscription or other 20 dollar subscriptions? Do you like the privacy?
  1. rnd0 · 1791186383 · focus · HN ↗
    Access to models is currently being priced below it's actual value. Largely because the market hasn't settled yet. Once it does the prices to access AI will increase -signifigantly.
    It will be less of an issue for people who have already got the equipment and the practice running locally than it will be for people who have simply been relying on ChatGPT.
    jeremyjh · 1791196573 · focus · HN ↗
    Prices on openrouter are not subsidized and are very affordable.

Unofficial Hacker News client; not affiliated with Y Combinator.