Thread

1. a11r · 1791132704 · focus · HN ↗

   I&#x27;m a little skeptical of going below 4-bit quants due to the potential for significant degradation in quality. I&#x27;m running 4-bit quants on an RTX Pro 6000 rented for approximately $1&#x2F;hour and getting about 1.2 million tokens out and 40 million tokens in per hour with caching. The quality of 4-bit quant is good enough for difficult but well-scoped coding tasks. Here is the inference stack I am using: <a href="https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;BlackwellPerformance&#x2F;s&#x2F;FrKwk3GoDK" rel="nofollow">https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;BlackwellPerformance&#x2F;s&#x2F;FrKwk3GoDK

   1. sfifs · 1791151727 · focus · HN ↗

      I&#x27;ve run DeepSeek V4 Flash on DS4 on single DGX and standard model weights on aDGX cluster. There was some degradation going to the hybrid 2 but quant but really not much.

Unofficial Hacker News client; not affiliated with Y Combinator.