Thread

Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

793 points · 354 comments · snehesht

a11r · 1791132704 · focus · HN ↗
I'm a little skeptical of going below 4-bit quants due to the potential for significant degradation in quality. I'm running 4-bit quants on an RTX Pro 6000 rented for approximately $1/hour and getting about 1.2 million tokens out and 40 million tokens in per hour with caching. The quality of 4-bit quant is good enough for difficult but well-scoped coding tasks. Here is the inference stack I am using: <a href="https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK" rel="nofollow">https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK
1. NinjaTrance · 1791137818 · focus · HN ↗
  Just as curiosity, how long does it take to set the environment up and running?
  Is it viable to start/stop it multiple times per day?
  1. a11r · 1791141893 · focus · HN ↗
    Yes, you can probably get the whole thing up and running in about an hour the first time. If you pause and restart, it takes about 15 minutes to load the models from disk into GPU memory, so budget for cold startup time.
    KeplerBoy · 1791143321 · focus · HN ↗
    How does it take fifteen minutes to read <100 GB into GPU memory? Shouldn't that be limited by SSD speed with everything slower than a minute being a terrible ssd?
    boredatoms · 1791148295 · focus · HN ↗
    It also depends on the runtime, vllm is unbelievably slow at model loading compared to llama.cpp
    ycui7 · 1791174744 · focus · HN ↗
    this is not right. vllm can easily load safetensors model at >5GB/s if not faster when setup right. did you make the compile cache persist? if you use docker, you should bind mount the kernel compile cache, so they don't need to be recompiled each time vllm restart.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.