"Nerf"ing models isn't real in the vast majority of reported cases. Benchmarks like this or the 100 other "let's see if nerfing is real" copies would have shown it by now if it was.
I made a graphic to explain why people feel like the models get nerfed:
The idea is that new models can handle up to a certain level of complexity, at which point they fall apart. Every new model can handle more complexity, so there's a wonderful time upon release when you feel like you can do anything, only for you to hit the complexity ceiling a few days later when you saturate it. Rinse and repeat for the next model.
If you click through to [1] that seems like a clear downwards trend (beyond the usual noise) about two weeks before the release of Opus 4.7, Opus 4.8, and Opus 5.5. Opus 5 is the only launch that looks clean without the previous model being nerfed beforehand
Sure, maybe it isn't the model getting nerved but the harness getting updates that make it better with the new model but substantially worse with the old (at that point still current) model.
The test doesn't differentiate. But neither can the average user, who will also be using the normal auto-updating harness. You still get degrading quality right before each new release
Yes, but then the model wasn't nerfed, the harness/overall product just had a plain old regression.
This is very different from a nefarious inference-side degradation to save cost, promote the new model or anything else frequently proposed as motivation.
johnfn · · focus · HN ↗
I made a graphic to explain why people feel like the models get nerfed:
<a href="https://x.com/thesilenceturns/status/2103551351825543610" rel="nofollow">https://x.com/thesilenceturns/status/2103551351825543610
The idea is that new models can handle up to a certain level of complexity, at which point they fall apart. Every new model can handle more complexity, so there's a wonderful time upon release when you feel like you can do anything, only for you to hit the complexity ceiling a few days later when you saturate it. Rinse and repeat for the next model.
scrollop · · focus · HN ↗
<a href="https://marginlab.ai/trackers/claude-code/" rel="nofollow">https://marginlab.ai/trackers/claude-code/
This site has been documenting it for a while
cbg0 · · focus · HN ↗
wongarsu · · focus · HN ↗
<a href="https://marginlab.ai/trackers/claude-code-historical-performance/" rel="nofollow">https://marginlab.ai/trackers/claude-code-historical-perform...
lxgr · · focus · HN ↗
> We always use the latest available Claude Code release and the SOTA model (currently Opus 5.5).
Changing the harness can have a big impact on performance even when leaving the model completely unchanged.
wongarsu · · focus · HN ↗
The test doesn't differentiate. But neither can the average user, who will also be using the normal auto-updating harness. You still get degrading quality right before each new release
lxgr · · focus · HN ↗
This is very different from a nefarious inference-side degradation to save cost, promote the new model or anything else frequently proposed as motivation.