"Nerf"ing models isn't real in the vast majority of reported cases. Benchmarks like this or the 100 other "let's see if nerfing is real" copies would have shown it by now if it was.
I made a graphic to explain why people feel like the models get nerfed:
The idea is that new models can handle up to a certain level of complexity, at which point they fall apart. Every new model can handle more complexity, so there's a wonderful time upon release when you feel like you can do anything, only for you to hit the complexity ceiling a few days later when you saturate it. Rinse and repeat for the next model.
There are recorded cases of real regressions, but they're better characterised as incidents, not nerfs, e.g.: <a href="https://www.anthropic.com/engineering/april-23-postmortem" rel="nofollow">https://www.anthropic.com/engineering/april-23-postmortem
Btw, you have a typo in the twitter handle on your profile (not in your comment), 'thesilencesturns'.
johnfn · · focus · HN ↗
I made a graphic to explain why people feel like the models get nerfed:
<a href="https://x.com/thesilenceturns/status/2103551351825543610" rel="nofollow">https://x.com/thesilenceturns/status/2103551351825543610
The idea is that new models can handle up to a certain level of complexity, at which point they fall apart. Every new model can handle more complexity, so there's a wonderful time upon release when you feel like you can do anything, only for you to hit the complexity ceiling a few days later when you saturate it. Rinse and repeat for the next model.
gobdovan · · focus · HN ↗
Btw, you have a typo in the twitter handle on your profile (not in your comment), 'thesilencesturns'.
johnfn · · focus · HN ↗