Thread

Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

741 points · 330 comments · snehesht

snehesht · 1791118313 · focus · HN ↗
I tried it and it worked surprisingly well. On my machine (Nvidia 4090, 128GB DDR5, Ryzen 7950x3d) I'm getting 124 tokens per sec, thought to share it here.
<a href="https://huggingface.co/Qwen/Qwen3.8-Flash-Next" rel="nofollow">https://huggingface.co/Qwen/Qwen3.8-Flash-Next
1. proc0 · 1791119012 · focus · HN ↗
  Do you know how it compares to Qwen 3.8 27B? I really want to compare the distilled ones with harness versus the full MoE versions.
  1. incognito124 · 1791120233 · focus · HN ↗
    Qwen 3.8 flash next is way better than 27B. It's so good I dont even use claude anymore
    JokerDan · 1791126377 · focus · HN ↗
    Is this true for 27b Q4_K_XL vs flash next IQ3_S? I thought under Q4 models start quickly degrading?
    gruturo · 1791132051 · focus · HN ↗
    While this is generally true, it's _a little_ less true the larger the model is.
    Also, quantization techniques have improved - the I in IQ3 stands for imatrix - Importance Matrix - it is a bit more surgical in what it cuts. The result is a model where the most important weights are even Q6 or above, the least important Q2 or even below, overall it takes the space of a Q3 but with better results.
    Tade0 · 1791142380 · focus · HN ↗
    To add to the other comment, there's also Ridge quantisation - the majority of weights are indeed Q3_x, but the most sensitive layers are FP8.
    latentsea · 1791164677 · focus · HN ↗
    This is the conventional wisdom, but in practice what matters is how reliably the model performs on your tasks in the real world. I have an R9700 and an RTX 5060 Ti and I've been running an IQ3_S quant of 27B on the 5060 Ti vs a Q6 quant on the R9700. I still manage to get stuff done with the IQ3_S quant.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.