Thread · HN Continuity

Self-optimizing inference is one of those problems where marginal gains compound—small improvements in batching, KV-cache reuse, and routing across thousands of requests add up fast.

The hard part isn't the optimization itself—it's measuring whether a change actually helps across the long tail of request patterns. Most inference benchmarks show huge gains on popular workloads, but production traffic has a fat tail where naive optimizations hurt latency.

Interested in how you handle regression detection when the optimizer changes between requests.