Thread

Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

741 points · 330 comments · snehesht

a11r · 1791132704 · focus · HN ↗
I'm a little skeptical of going below 4-bit quants due to the potential for significant degradation in quality. I'm running 4-bit quants on an RTX Pro 6000 rented for approximately $1/hour and getting about 1.2 million tokens out and 40 million tokens in per hour with caching. The quality of 4-bit quant is good enough for difficult but well-scoped coding tasks. Here is the inference stack I am using: <a href="https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK" rel="nofollow">https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK
1. sail0rm00n · 1791132917 · focus · HN ↗
  $1/hr sounds great. Where are you getting it for those prices?
  1. transcriptase · 1791134049 · focus · HN ↗
    [flagged]
    xingped · 1791135709 · focus · HN ↗
    No? Per someone else's comment, presumably they're using vast.ai which does indeed list the 6000 for $1/hr
    dang · 1791140307 · focus · HN ↗
    Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for.
    If you wouldn't mind reviewing <a href="https://news.ycombinator.com/newsguidelines.html">https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
    transcriptase · 1791143870 · focus · HN ↗
    Apologies. I had recently read (I think from someone on here) that they were in some discord where those sellers advertise. Whoever it was mentioned that the service was incredible value for what they got, but admittedly it was for toy projects, there was no written agreements, and uptime was something like 98%. I should have found the source and linked, and will try to do better overall in my replies!
2. robot_jesus · 1791132925 · focus · HN ↗
  Can you say more about where you're renting the RTX Pro 6000 for $1/hour?
  1. _zoltan_ · 1791135272 · focus · HN ↗
    vast.ai?
  2. a11r · 1791141918 · focus · HN ↗
    I am renting spot VMs from Nebius. I've tried a variety of other providers like Vast and Spheron. Vast worked well for renting 5090s but I like the large memory and pricing I'm getting at Nebius for RTX Pro 6000. The extra RAM really matters because I need PLE to offload the ngram to RAM.
3. aatd86 · 1791133797 · focus · HN ↗
  $1/hour ? I need that deal as well.
  1. trvz · 1791134289 · focus · HN ↗
    Not the original commenter, but Upcloud has them at that price for spot instances: <a href="https://upcloud.com/pricing/" rel="nofollow">https://upcloud.com/pricing/
4. NinjaTrance · 1791137818 · focus · HN ↗
  Just as curiosity, how long does it take to set the environment up and running?
  Is it viable to start/stop it multiple times per day?
  1. a11r · 1791141893 · focus · HN ↗
    Yes, you can probably get the whole thing up and running in about an hour the first time. If you pause and restart, it takes about 15 minutes to load the models from disk into GPU memory, so budget for cold startup time.
    KeplerBoy · 1791143321 · focus · HN ↗
    How does it take fifteen minutes to read <100 GB into GPU memory? Shouldn't that be limited by SSD speed with everything slower than a minute being a terrible ssd?
    teaearlgraycold · 1791143482 · focus · HN ↗
    A lot of cloud platforms have terrible slow network storage. They also might need to compile the GPU kernels fresh as they might not have a persistent CUDA cache.
    boredatoms · 1791148295 · focus · HN ↗
    It also depends on the runtime, vllm is unbelievably slow at model loading compared to llama.cpp
    ycui7 · 1791174744 · focus · HN ↗
    this is not right. vllm can easily load safetensors model at >5GB/s if not faster when setup right. did you make the compile cache persist? if you use docker, you should bind mount the kernel compile cache, so they don't need to be recompiled each time vllm restart.
  2. prettyblocks · 1791170157 · focus · HN ↗
    I just had claude code set it up for me and create some startup scripts. The longest part was the model download.
5. nialv7 · 1791144153 · focus · HN ↗
  ISTA-DASLab's IQ3_S quant is really good <a href="https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF" rel="nofollow">https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RC...
  1. ricardobeat · 1791144380 · focus · HN ↗
    What are you running it on? I'm getting mixed results using the IQ3_XXS quant which supposedly matches baseline, it feels significantly degraded.
6. nickpsecurity · 1791145097 · focus · HN ↗
  That's cheap, too. Which hosting service are you using?
7. Winfred-zz · 1791149640 · focus · HN ↗
  I just ran a set of benchmarks, ninfer-3090-qwen3.8-27b (so mix of Q4 and Q5) vs strata-qwen3.8-flash-next-iq3_xxs (so Q3):
  │------------------- │ Ninfer-3090 │ Strata
  │ Code generation │ 52/78 (66.7%) │ 70/78 (89.7%)
  │ Code completion │ 40/50 (80.0%) │ 44/50 (88.0%)
  │ Total------------- │ 92/128 (71.9%) │ 114/128 (89.1%)
  │ API failures------ │ 10 │ 5
  - Ninfer generation: ~122 min total.
  - Strata generation: ~142 min total.
  So strata is a little slower, but keep in mind that ninfer-3090 is very optimized for a Qwen 3.8. Standard Qwen 3.8 runs at 20 t/s, this modified version can do 50 t/s (but it's extremely long in it's thinking, it just goes on and on.
  This is on a 3090 that will crash unless power capped, with a Zen 2 CPU, 64GB DDR4 with a PCIe that refuses to go higher than 8x (basically pretty crappy all in all).
  Yet with some tweaking and optimizing I still manage to get strata to run at 40 to 60 t/s.
  That strata has been optimized on my Oh My Pi conversations. So when I'm using it, it's probably faster and closer to ninfer in speed than during those unoptimized benchmark tests.
  1. sleight42 · 1791153578 · focus · HN ↗
    Anecdotally, I've found that 27B at 4bit hallucinates a lot more than QFN at 3_xxs, particularly for less technical reasoning.
    I've tried to use both for online comparison shopping. QFN not only seemed less delusional but also made useful observations and problem solved ways around many different website access issues.
  2. tharkun__ · 1791164765 · focus · HN ↗
    You can use 4 spaces on HN to get code/monospaced formatting:
    │------------------- │ Ninfer-3090 │ Strata │ Code generation │ 52/78 (66.7%) │ 70/78 (89.7%) │ Code completion │ 40/50 (80.0%) │ 44/50 (88.0%) │ Total------------- │ 92/128 (71.9%) │ 114/128 (89.1%) │ API failures------ │ 10 │ 5
  3. greenavocado · 1791167145 · focus · HN ↗
    In my private C coding benchmark, Strata's Qwen Flash Next Coder IQ1_M beats Luna low and runs at 125 tok/s on my 5090
  4. airspresso · 1791179140 · focus · HN ↗
    > it's extremely long in it's thinking, it just goes on and on
    Known issue with this model, I recommend setting thinking to 'medium' instead of default 'xhigh'.
  5. seviu · 1791179791 · focus · HN ↗
    Unrelated... just use your just locally installed qwen 3.8 and instruct it to figure out why you arent at 16x.
    For me, I didnt know I was on 4x and I was able to, giving it enough permissions, with a good harness like PI, to figure things out and go 4x -> 8x -> 16x.
    It probably will take some opening the case and switching ssds around, but it's amazing how powerful these models have become.
8. sfifs · 1791151727 · focus · HN ↗
  I've run DeepSeek V4 Flash on DS4 on single DGX and standard model weights on aDGX cluster. There was some degradation going to the hybrid 2 but quant but really not much.
9. segmondy · 1791153051 · focus · HN ↗
  The larger the model, the more you can go down. K3 in Q1 will match and likely beat Qwen3.8-Flash-Next.
10. sudo_cowsay · 1791159572 · focus · HN ↗
  For the 1 dollar per hour, may I ask (without trying to incite things) why you would not just buy a ChatGPT Plus subscription or other 20 dollar subscriptions? Do you like the privacy?
  1. mannanj · 1791163200 · focus · HN ↗
    maybe ethics as well. maybe people like sleeping well at night.
    tfrancisl · 1791165223 · focus · HN ↗
    [This post has been removed for discussing politics.]
  2. manmal · 1791179773 · focus · HN ↗
    Those plans are super slow right now. Nowhere near 100t/s.
    martianvoid · 1791180143 · focus · HN ↗
    I think Opus 5.5 is now fast enough with the inference stack improvements that they did. Sonnet 5.5 is around 140 tokens/s when I had measured it last time. A $20 subs on claude and using only sonnet 5.5 will last a lot
11. alvarolucero · 1791182904 · focus · HN ↗
  [dead]

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.