>"CLM-8B also sets a new SOTA on challenging agentic coding benchmarks, including DeepSWE (81.6%) and Terminal Bench 2.1 (87.6%).
I didn't see any details on this on the announce page. And I don't believe it. Astra x-high pass@1 on DeepSWE is 74% +/- 3%. (<a href="https://deepswe.datacurve.ai">https://deepswe.datacurve.ai).
That said, love seeing some of these new architectures get people exploring. But, surely somebody is incorrect here inre: those numbers.
They don't report the pass@1 success rate. They sample multiple solutions from Opus/Fable and CLM decides which one to submit, that's why they get >80%.
Maybe I'm misunderstanding this but when would you ever use it this way? If you're already willing to call Opus/Fable, then isn't the obvious comparison whether Opus/Fable can choose among sampled solutions better or worse than their fast model? If you're willing to pay many seconds for many code samples from a slow model, it's contrived to imagine you care about picking between them in ms.
vessenes · · focus · HN ↗
I didn't see any details on this on the announce page. And I don't believe it. Astra x-high pass@1 on DeepSWE is 74% +/- 3%. (<a href="https://deepswe.datacurve.ai">https://deepswe.datacurve.ai).
That said, love seeing some of these new architectures get people exploring. But, surely somebody is incorrect here inre: those numbers.
kuukyo · · focus · HN ↗
abeppu · · focus · HN ↗