Thread

Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

819 points · 362 comments · snehesht

a11r · 1791132704 · focus · HN ↗
I'm a little skeptical of going below 4-bit quants due to the potential for significant degradation in quality. I'm running 4-bit quants on an RTX Pro 6000 rented for approximately $1/hour and getting about 1.2 million tokens out and 40 million tokens in per hour with caching. The quality of 4-bit quant is good enough for difficult but well-scoped coding tasks. Here is the inference stack I am using: <a href="https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK" rel="nofollow">https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK
1. sudo_cowsay · 1791159572 · focus · HN ↗
  For the 1 dollar per hour, may I ask (without trying to incite things) why you would not just buy a ChatGPT Plus subscription or other 20 dollar subscriptions? Do you like the privacy?
  1. mannanj · 1791163200 · focus · HN ↗
    maybe ethics as well. maybe people like sleeping well at night.
    tfrancisl · 1791165223 · focus · HN ↗
    [This post has been removed for discussing politics.]
  2. manmal · 1791179773 · focus · HN ↗
    Those plans are super slow right now. Nowhere near 100t/s.
    martianvoid · 1791180143 · focus · HN ↗
    I think Opus 5.5 is now fast enough with the inference stack improvements that they did. Sonnet 5.5 is around 140 tokens/s when I had measured it last time. A $20 subs on claude and using only sonnet 5.5 will last a lot
    c16 · 1791185067 · focus · HN ↗
    I was firmly wanting to move towards local llm, and since the Sonnet 5.5 and recent Opus speed increases, I'm putting that on hold until local inference speeds can be improved. Not sure how much juice there is to squeeze there, but I'm hopeful.
    azath92 · 1791187685 · focus · HN ↗
    While this may be true, their throughput is practically unbounded. this has been a primary challenge to using local models at work, where its normal for me to have 1-6 sessions churning away at once. So while 100t/s on a single session is very fast, theres no way you could get a cumulative ~500t/s from a consumer setup like this, do the bandwitdth bottleneck on a single card/cpu/ram/ssd (tends to be gpu ram bandwidth is the limiter in my setup, but they all have respective limiters to this kind of throughput)
    Almondsetat · 1791189408 · focus · HN ↗
    Does it matter if those 100 tokens per second are way crappier?
  3. rnd0 · 1791186383 · focus · HN ↗
    Access to models is currently being priced below it's actual value. Largely because the market hasn't settled yet. Once it does the prices to access AI will increase -signifigantly.
    It will be less of an issue for people who have already got the equipment and the practice running locally than it will be for people who have simply been relying on ChatGPT.
    jeremyjh · 1791196573 · focus · HN ↗
    Prices on openrouter are not subsidized and are very affordable.
  4. bad_haircut72 · 1791186733 · focus · HN ↗
    People think claude costs $200/month, it actually costs $200/month and all your labor to train their models, improve their product, and all your best ideas and work that they will feed back into their company. Its very expensive
    komatar · 1791191172 · focus · HN ↗
    You can opt out of model training with a checkbox, though I doubt most companies respect this completely. At the very least, I imagine they find loopholes to exploit.
    But on the other hand I'm giving them my best and worst ideas and the value I get in return outweighs what I'm contributing to them. I also don't have the problems of initial HW cost and maintenance.
    radicalbyte · 1791194571 · focus · HN ↗
    > You can opt out of model training with a checkbox
    Do you trust that a company who can't even do basic IT ops monitoring are capable of respecting a "no training checkbox"?
    wren6991 · 1791195965 · focus · HN ↗
    Great, then they will PII-scrub my sessions before feeding them into the training pool :-)

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.