Thread

Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

806 points · 356 comments · snehesht

a11r · 1791132704 · focus · HN ↗
I'm a little skeptical of going below 4-bit quants due to the potential for significant degradation in quality. I'm running 4-bit quants on an RTX Pro 6000 rented for approximately $1/hour and getting about 1.2 million tokens out and 40 million tokens in per hour with caching. The quality of 4-bit quant is good enough for difficult but well-scoped coding tasks. Here is the inference stack I am using: <a href="https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK" rel="nofollow">https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK
1. Winfred-zz · 1791149640 · focus · HN ↗
  I just ran a set of benchmarks, ninfer-3090-qwen3.8-27b (so mix of Q4 and Q5) vs strata-qwen3.8-flash-next-iq3_xxs (so Q3):
  │------------------- │ Ninfer-3090 │ Strata
  │ Code generation │ 52/78 (66.7%) │ 70/78 (89.7%)
  │ Code completion │ 40/50 (80.0%) │ 44/50 (88.0%)
  │ Total------------- │ 92/128 (71.9%) │ 114/128 (89.1%)
  │ API failures------ │ 10 │ 5
  - Ninfer generation: ~122 min total.
  - Strata generation: ~142 min total.
  So strata is a little slower, but keep in mind that ninfer-3090 is very optimized for a Qwen 3.8. Standard Qwen 3.8 runs at 20 t/s, this modified version can do 50 t/s (but it's extremely long in it's thinking, it just goes on and on.
  This is on a 3090 that will crash unless power capped, with a Zen 2 CPU, 64GB DDR4 with a PCIe that refuses to go higher than 8x (basically pretty crappy all in all).
  Yet with some tweaking and optimizing I still manage to get strata to run at 40 to 60 t/s.
  That strata has been optimized on my Oh My Pi conversations. So when I'm using it, it's probably faster and closer to ninfer in speed than during those unoptimized benchmark tests.
  1. sleight42 · 1791153578 · focus · HN ↗
    Anecdotally, I've found that 27B at 4bit hallucinates a lot more than QFN at 3_xxs, particularly for less technical reasoning.
    I've tried to use both for online comparison shopping. QFN not only seemed less delusional but also made useful observations and problem solved ways around many different website access issues.
  2. tharkun__ · 1791164765 · focus · HN ↗
    You can use 4 spaces on HN to get code/monospaced formatting:
    │------------------- │ Ninfer-3090 │ Strata │ Code generation │ 52/78 (66.7%) │ 70/78 (89.7%) │ Code completion │ 40/50 (80.0%) │ 44/50 (88.0%) │ Total------------- │ 92/128 (71.9%) │ 114/128 (89.1%) │ API failures------ │ 10 │ 5
  3. greenavocado · 1791167145 · focus · HN ↗
    In my private C coding benchmark, Strata's Qwen Flash Next Coder IQ1_M beats Luna low and runs at 125 tok/s on my 5090
    seviu · 1791187379 · focus · HN ↗
    Not as fast for me. I was able to snatch a spark before the horrible price hike, and I have it at 40/50tok/sec. Before I had the great 3090 setup which is what led me to get the spark. That 3090 is now relegated to creating videos of my kids doing dumb things.
    I now backordered two more sparks, before the hike... hoping they will honor the agreed price and dont cancel on me. They should arrive in a month,
    Its evidently clear the frontier labs won't keep on giving us cheap inference for much longer. I also watch in disbelief how people say a sub is cheaper, which probably is. But you loose on so many other things (privacy, predictability, censorship...)
  4. airspresso · 1791179140 · focus · HN ↗
    > it's extremely long in it's thinking, it just goes on and on
    Known issue with this model, I recommend setting thinking to 'medium' instead of default 'xhigh'.
  5. seviu · 1791179791 · focus · HN ↗
    Unrelated... just use your just locally installed qwen 3.8 and instruct it to figure out why you arent at 16x.
    For me, I didnt know I was on 4x and I was able to, giving it enough permissions, with a good harness like PI, to figure things out and go 4x -> 8x -> 16x.
    It probably will take some opening the case and switching ssds around, but it's amazing how powerful these models have become.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.