"Nerf"ing models isn't real in the vast majority of reported cases. Benchmarks like this or the 100 other "let's see if nerfing is real" copies would have shown it by now if it was.
I made a graphic to explain why people feel like the models get nerfed:
The idea is that new models can handle up to a certain level of complexity, at which point they fall apart. Every new model can handle more complexity, so there's a wonderful time upon release when you feel like you can do anything, only for you to hit the complexity ceiling a few days later when you saturate it. Rinse and repeat for the next model.
Anthropic has admitted to nerfing in the past. There have also been inference bugs. On top of that, model performance changes as they move compute to schwaggier providers as well.
does it really matter wherever its the harness or the model for the vibe coder user claude code?
fwiw, i think almost all regressions are down to a/b testing in the harness by anthropic, but it is objectively indistinguishable beyond "the coding agent ceases to be usable" and i'm back to traditional coding for a few hours until its back to normal again
It absolutely matters because something like Claude Code has no guarantee that there won't be changes between updates but a model pinned at the API version level that is getting enterprise traffic absolutely does have that guarantee and would be a much more widespread problem...
johnfn · · focus · HN ↗
I made a graphic to explain why people feel like the models get nerfed:
<a href="https://x.com/thesilenceturns/status/2103551351825543610" rel="nofollow">https://x.com/thesilenceturns/status/2103551351825543610
The idea is that new models can handle up to a certain level of complexity, at which point they fall apart. Every new model can handle more complexity, so there's a wonderful time upon release when you feel like you can do anything, only for you to hit the complexity ceiling a few days later when you saturate it. Rinse and repeat for the next model.
prodigycorp · · focus · HN ↗
Anthropic has admitted to nerfing in the past. There have also been inference bugs. On top of that, model performance changes as they move compute to schwaggier providers as well.
Your chart is wrong.
simonw · · focus · HN ↗
Where?
QwenGlazer9000 · · focus · HN ↗
Unintentional tbf.
weird-eye-issue · · focus · HN ↗
ffsm8 · · focus · HN ↗
fwiw, i think almost all regressions are down to a/b testing in the harness by anthropic, but it is objectively indistinguishable beyond "the coding agent ceases to be usable" and i'm back to traditional coding for a few hours until its back to normal again
weird-eye-issue · · focus · HN ↗
It absolutely matters because something like Claude Code has no guarantee that there won't be changes between updates but a model pinned at the API version level that is getting enterprise traffic absolutely does have that guarantee and would be a much more widespread problem...