Thread

Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

741 points · 330 comments · snehesht

fsiefken · 1791127208 · focus · HN ↗
I wonder if a higher Qwen3.8-27b quant could beat or match these lower < 16/24/48/64G Qwen3.8-Flash Next quants given similar quality.
What speed are you willing the sacrifice to debug/program for more complex jobs faster?
Then there are also these quants; <a href="https://huggingface.co/IsValorum/Qwen3.8-35B-A3B-Distill-MLX-APEX-NanoPlus-Abliterated" rel="nofollow">https://huggingface.co/IsValorum/Qwen3.8-35B-A3B-Distill-MLX...
1. xreborn · 1791127358 · focus · HN ↗
  from my experience dense models like 27b suffer less from quantization compared to large MoEs
2. zkmon · 1791127699 · focus · HN ↗
  I'm not going to knock off my 27B-Q_6 for this. Good to to experiment though.
3. latentsea · 1791130664 · focus · HN ↗
  The benchmark indicates the IQ3_XXS quant beats 27B. I've switched to that now and am ditching 27B. Genuinely better results so far.
4. happycube · 1791143521 · focus · HN ↗
  Maybe, but those higher quants would need a large GPU accessible memory space - and the obvious candidates such as DGX Spark and Strix Halo don't have the bandwidth to run 27B at high quality quickly.
  With Flash Next you only have ~6B active parameters so you can toss experts up into VRAM and/or run them on a CPU if you have enough RAM and bandwidth.

Unofficial Hacker News client; not affiliated with Y Combinator.