Thread

Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

741 points · 330 comments · snehesht

mmaunder · 1791125750 · focus · HN ↗
More great work on local model but you’re still losing a lot. Down to 2 bit quantization and the coder model throws away half the MoE experts. In a world where anything is better than nothing, this is a net win. But we have a way to go still.
1. latentsea · 1791130497 · focus · HN ↗
  You can run IQ3_XXS, IQ3_S and the IQ4_XS quants on this too. It works. It's fantastic. I'm getting better results than 27B now.
  1. bitexploder · 1791131223 · focus · HN ↗
    To add: <a href="https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF" rel="nofollow">https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RC... IQ3_XXS is within a point of the fully unquantized model and IQ3_S actually beats the unquantized model on many tasks! You lose absolutely nothing. It is quantization magic :)
    Muromec · 1791158717 · focus · HN ↗
    No magic needed, the quantization found the expert which deals in java and deleted it, so the model overall became better.
    bitexploder · 1791159076 · focus · HN ↗
    lol. JDQ -- Java Delete Quantization it's the new thing!
    latentsea · 1791163764 · focus · HN ↗
    Ah, I see you've found the AbstractExpertRemovalFactoryFactory!
2. Culonavirus · 1791135470 · focus · HN ↗
  The real problem is the DRAM mafia and artificial scarcity. One of my notebooks is almost 3 years old, effectively similar spec now - same price (a bit higher actually). My desktop PC built around march/april 2023 (4090, 64gb ram, 7900x3d) is now pretty much still the top dog out there due to the gpu and fast ram insanity and if I wanted to sell it today, I'd get more money for it now used and over 3 years old that when I bought it!
  We should be having 64/72+ GB video cards by now. 128GB+ system ram prosumer laptops and 256GB+ system ram prosumer/gamer desktops. But it all went to shit and it will require some brutal datacenter and datacenter-adjacent bankruptcies before it gets better.
  Some of these greedy bastards need to lose their pants on all of this.
  1. RachelF · 1791160067 · focus · HN ↗
    This is true, progress has stopped on the RAM and VRAM front.
    Many new laptops come with 8GB as standard, the same as 12 years ago.
    My 1060 from 2016 has 6GB of VRAM. A 5060 from 2025 has 8GB.
  2. dboreham · 1791166605 · focus · HN ↗
    Memory DIMMs could be the Aeron chair of the 2030s.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.