Thread

Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

826 points · 365 comments · snehesht

a11r · 1791132704 · focus · HN ↗
I'm a little skeptical of going below 4-bit quants due to the potential for significant degradation in quality. I'm running 4-bit quants on an RTX Pro 6000 rented for approximately $1/hour and getting about 1.2 million tokens out and 40 million tokens in per hour with caching. The quality of 4-bit quant is good enough for difficult but well-scoped coding tasks. Here is the inference stack I am using: <a href="https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK" rel="nofollow">https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK
1. sudo_cowsay · 1791159572 · focus · HN ↗
  For the 1 dollar per hour, may I ask (without trying to incite things) why you would not just buy a ChatGPT Plus subscription or other 20 dollar subscriptions? Do you like the privacy?
  1. bad_haircut72 · 1791186733 · focus · HN ↗
    People think claude costs $200/month, it actually costs $200/month and all your labor to train their models, improve their product, and all your best ideas and work that they will feed back into their company. Its very expensive
    komatar · 1791191172 · focus · HN ↗
    You can opt out of model training with a checkbox, though I doubt most companies respect this completely. At the very least, I imagine they find loopholes to exploit.
    But on the other hand I'm giving them my best and worst ideas and the value I get in return outweighs what I'm contributing to them. I also don't have the problems of initial HW cost and maintenance.
    radicalbyte · 1791194571 · focus · HN ↗
    > You can opt out of model training with a checkbox
    Do you trust that a company who can't even do basic IT ops monitoring are capable of respecting a "no training checkbox"?

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.