Thread

1. johnfn · 1790724998 · focus · HN ↗

   "Nerf"ing models isn't real in the vast majority of reported cases. Benchmarks like this or the 100 other "let's see if nerfing is real" copies would have shown it by now if it was.

   I made a graphic to explain why people feel like the models get nerfed:

   <a href="https:&#x2F;&#x2F;x.com&#x2F;thesilenceturns&#x2F;status&#x2F;2103551351825543610" rel="nofollow">https:&#x2F;&#x2F;x.com&#x2F;thesilenceturns&#x2F;status&#x2F;2103551351825543610

   The idea is that new models can handle up to a certain level of complexity, at which point they fall apart. Every new model can handle more complexity, so there&#x27;s a wonderful time upon release when you feel like you can do anything, only for you to hit the complexity ceiling a few days later when you saturate it. Rinse and repeat for the next model.

   1. prodigycorp · 1790726005 · focus · HN ↗

      Incorrect.

      Anthropic has admitted to nerfing in the past. There have also been inference bugs. On top of that, model performance changes as they move compute to schwaggier providers as well.

      Your chart is wrong.

      1. simonw · 1790726139 · focus · HN ↗

         &gt; Anthropic has admitted to nerfing in the past

         Where?

         1. prodigycorp · 1790726368 · focus · HN ↗

            Man this was last year and some Claude subreddit drama that I can’t furnish offf the top of my head but maybe one of the historians remember it.

            1. p-e-w · 1790727942 · focus · HN ↗

               Noone can seem to remember anything with certainty when asked to actually substantiate these claims.

               1. erinnh · 1790728732 · focus · HN ↗

                  I mean there is a direct link two comments down from here from 30 minutes before your comment: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=49902477">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=49902477

                  1. p-e-w · 1790733143 · focus · HN ↗

                     That’s NOT Anthropic admitting to “nerfing” their model as claimed above (which implies intent), that’s a regression which they quickly fixed.

                     Christ this forum has become intellectually dishonest.

                     1. jibalt · 1790755025 · focus · HN ↗

                        There has always been plenty of intellectual dishonesty here, but remember Hanlon&#x27;s Razor.
                     2. [deleted] · 1790773263 · focus · HN ↗

                        [deleted]

                  2. jibalt · 1790755049 · focus · HN ↗

                     Er, perhaps take a look after the edit:

                     &gt; There are recorded cases of real regressions, but they&#x27;re better characterised as incidents, not nerfs

                     1. erinnh · 1790765606 · focus · HN ↗

                        I dont really characterize a nerf as always by intent.

                        So I found this incident, as they called it, to still be relevant and why benchmarks such as the OP are useful.

Unofficial Hacker News client; not affiliated with Y Combinator.