Thread

Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

819 points · 362 comments · snehesht

snehesht · 1791118313 · focus · HN ↗
I tried it and it worked surprisingly well. On my machine (Nvidia 4090, 128GB DDR5, Ryzen 7950x3d) I'm getting 124 tokens per sec, thought to share it here.
<a href="https://huggingface.co/Qwen/Qwen3.8-Flash-Next" rel="nofollow">https://huggingface.co/Qwen/Qwen3.8-Flash-Next
1. proc0 · 1791119012 · focus · HN ↗
  Do you know how it compares to Qwen 3.8 27B? I really want to compare the distilled ones with harness versus the full MoE versions.
  1. thatsabadlook · 1791121654 · focus · HN ↗
    Significantly better for both performance and real world use case. 3.8 27b is a good small model. This is a good model.
    geye1234 · 1791122025 · focus · HN ↗
    I find 27B more accurate -- maybe because I'm running at FP8 instead of NVFP4? Flash Next starts making spelling mistakes when I get to 150K context or so. Also it sometimes ignores .md file instructions. Not sure if others have found that.
    PcChip · 1791123451 · focus · HN ↗
    Spelling mistakes?
    What inference engine are you using for flash next?
    geye1234 · 1791150089 · focus · HN ↗
    I'm running Pennyroyal's Docker image (on Podman) which uses sglang. I have a single RTX 6000 Blackwell and 128GB RAM. I turned off disk caching. I'm running with a ~500K context, but have been limiting it to 256K in the client (pi).
    It always detects its spelling mistakes, btw, but it worried me. It may turn 'rm -rf ' into 'rm -rf /' one day.
    Almost certainly the problem is my config, not the image.
    xiconfjs · 1791180684 · focus · HN ↗
    Did you check your GPU for memory errors/defects?

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.