More benchmark noise. They’re already problematic per se, and the error bars only get bigger by aggregating and adding costs on top of it.
Create your own, private evaluation system that reflects your use and your constraints. Focus on hard cases you encountered, find cases that break on one tier and not on another.
Exactly, according to these benchmarks Gemini 3.8 Flash is the best option at that budget level for coding, but in real world testing, I found DeepSeek flash to perform far better at a much lower price.
khalic · · focus · HN ↗
Create your own, private evaluation system that reflects your use and your constraints. Focus on hard cases you encountered, find cases that break on one tier and not on another.
krmmalik · · focus · HN ↗
Firaxus · · focus · HN ↗