Thread

Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

741 points · 330 comments · snehesht

mmaunder · 1791125750 · focus · HN ↗
More great work on local model but you’re still losing a lot. Down to 2 bit quantization and the coder model throws away half the MoE experts. In a world where anything is better than nothing, this is a net win. But we have a way to go still.
1. latentsea · 1791130497 · focus · HN ↗
  You can run IQ3_XXS, IQ3_S and the IQ4_XS quants on this too. It works. It's fantastic. I'm getting better results than 27B now.
  1. bitexploder · 1791131223 · focus · HN ↗
    To add: <a href="https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF" rel="nofollow">https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RC... IQ3_XXS is within a point of the fully unquantized model and IQ3_S actually beats the unquantized model on many tasks! You lose absolutely nothing. It is quantization magic :)
    Muromec · 1791158717 · focus · HN ↗
    No magic needed, the quantization found the expert which deals in java and deleted it, so the model overall became better.
    bitexploder · 1791159076 · focus · HN ↗
    lol. JDQ -- Java Delete Quantization it's the new thing!
    latentsea · 1791163764 · focus · HN ↗
    Ah, I see you've found the AbstractExpertRemovalFactoryFactory!

Unofficial Hacker News client; not affiliated with Y Combinator.