Thread

Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

741 points · 330 comments · snehesht

a11r · 1791132704 · focus · HN ↗
I'm a little skeptical of going below 4-bit quants due to the potential for significant degradation in quality. I'm running 4-bit quants on an RTX Pro 6000 rented for approximately $1/hour and getting about 1.2 million tokens out and 40 million tokens in per hour with caching. The quality of 4-bit quant is good enough for difficult but well-scoped coding tasks. Here is the inference stack I am using: <a href="https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK" rel="nofollow">https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK
1. sail0rm00n · 1791132917 · focus · HN ↗
  $1/hr sounds great. Where are you getting it for those prices?
  1. transcriptase · 1791134049 · focus · HN ↗
    [flagged]
    xingped · 1791135709 · focus · HN ↗
    No? Per someone else's comment, presumably they're using vast.ai which does indeed list the 6000 for $1/hr
    dang · 1791140307 · focus · HN ↗
    Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for.
    If you wouldn't mind reviewing <a href="https://news.ycombinator.com/newsguidelines.html">https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
    transcriptase · 1791143870 · focus · HN ↗
    Apologies. I had recently read (I think from someone on here) that they were in some discord where those sellers advertise. Whoever it was mentioned that the service was incredible value for what they got, but admittedly it was for toy projects, there was no written agreements, and uptime was something like 98%. I should have found the source and linked, and will try to do better overall in my replies!

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.