For reference, Mimo-v2.5-Pro scored 19% on DeepSWE 1.1. This is looking great.Fable scores 70%, Kimi K3 69%, Astra 74% (all on max effort).<a href="https://deepswe.datacurve.ai/blog/deepswe-v1-1">https://deepswe.datacurve.ai/blog/deepswe-v1-1
2.6-pro just reached 63.7% by step 10, it's on step 11 right now.Even flash reached 60.7% by step 12, and it's on step 16 now.This is so exciting lmao.
DeepSWE is saturated now IMO, and is basically worthless. Lots of new models get around 74%. Shame too, because it was a pretty decent benchmark for a few months there.
It is saturated, but that doesn’t mean worthless. Seeing 72% is low-signal, but 30% is still meaningful.
ricardobeat · · focus · HN ↗
Fable scores 70%, Kimi K3 69%, Astra 74% (all on max effort).
<a href="https://deepswe.datacurve.ai/blog/deepswe-v1-1">https://deepswe.datacurve.ai/blog/deepswe-v1-1
Cookingboy · · focus · HN ↗
Even flash reached 60.7% by step 12, and it's on step 16 now.
This is so exciting lmao.
arcanemachiner · · focus · HN ↗
brookst · · focus · HN ↗