‹ BackHN Continuity

Thread

Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

816 points · 361 comments · snehesht

  1. quietFalcon · · focus · HN ↗
    Nice, though generation speed is the easy half for MoE offload, what's your prompt processing look like at say 16k context?
    1. merbanan · · focus · HN ↗
      Q2_0 does 33 tok/s decode and ~600t/s prompt processing at 128k context on RTX2060 8GB VRAM.

      ISTA IQ3_XXS does ~21 tok/s decode and ~240t/s prompt processing

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.