This is undoubtedly great. But most of the inference cost today for dominant use cases (agentic coding) are in the prefill, not the decode. This is one of the reasons that DeepSeek is so aggressively optimizing prefill and caching.
qeternity · · focus · HN ↗