They test it on launch day, then benchmark it against that. A deviation of above 10% is considered a change. They're currently tracking Opus 5.5 and GPT-6 Astra.
This bench famously detected a degradation of Opus 4.6 which Anthropic later blogged about. I personally think people sense nerfs more often than they happen and that it's often about honeymoon effects.
Used to work on a chat app where we had full control of the stack from the GPUs to the chat interface and everything in between. We were a small team too so I could be pretty confident that nothing changed in the stack and we still regularly had users complain that this or that model got nerfed. Perceived performance is actual performance over expectations and the latter just keeps increasing over time.
It doesn’t mean the big labs don’t also nerf models! But if they didn’t you’d still have users complaining.
For years I used to buy ASICS running shoes. Every year they released a new model of each shoe: "Nimbus 23", then "Nimbus 24" the next year, etc. And every year people would complain in the user reviews about how each shoe was worse than the last.
I was like, wow, I guess the shoes must be literal torture devices full of MRSA-covered broken glass at this point. They've been getting continuously worse for 24 consecutive years!
Of course, what was really happening is that they were not getting worse, but naturally every year there was some small percentage of vocal dissatisfied users, while the silent majority simply enjoyed their shoes and didn't have much to say about them.
(The sorta-opposite happens in sneaker reviews as well. People will gush about how cushy the sole in some particular new sneaker is. Well, yeah, of course it's cushy -- you're comparing a new sneaker to your old sneaker where the foam had lost its bounce...)
I wonder how much of that is caused by folks actually noticing actual degradation in product quality over time (whether or not this exact product is suffering from it).
Shrinkflation is a thing, which people suddenly started noticing in the past 5 years.
There's also "the Schlitz Mistake", which I've heard summarized as "most customers won't notice if you take your product's quality from A to B (or C), but they definitely will if you take it from A to J (or A to M)"
At this point I kinda assume that any company releasing year updates to a physical product that _doesn't_ take the opportunity to trim costs / reduce quality would be vulnerable to a shareholder lawsuit for leaving money on the table...
I'm curious how common such lawsuits are. I don't think I've heard of any specific instances where shareholders sued because a company didn't make the product worse.
On the other hand that idea that "companies exist to make money for shareholders, to ONLY make money for shareholders, and doing any other than maximizing shareholder returns is bad" is pretty commonly accepted (and, I believe, enshrined in US law)
> Shrinkflation is a thing, which people suddenly started noticing in the past 5 years.
It's been much longer than that: <a href="https://en.wikipedia.org/wiki/Toblerone#2016_size_changes" rel="nofollow">https://en.wikipedia.org/wiki/Toblerone#2016_size_changes
> "most customers won't notice if you take your product's quality from A to B (or C), but they definitely will if you take it from A to J (or A to M)"
> I wonder how much of that is caused by folks actually noticing actual degradation in product quality over time (whether or not this exact product is suffering from it).
That's certainly common!
I think ASICS' running shoes were a fun example of where this was probably not the case.
- I certainly didn't notice a difference in that time, though I'm admittedly not much of an actual runner
- Serious runners might notice small technical differences, but the negative user reviews didn't seem to indicate those were the people making the complaints
- The overall user reviews remained positive
- The competition in the shoe market is incredibly fierce; I'm not sure a brand could tank their quality and survive for long
- Let's not forget the other big variable: the wearers' bodies, particularly their feet. Now, those definitely do change over time -- most often for the worse, sadly!
- I doubt anybody was blind A/Bing a pair of Nimbus 20 against a pair of Nimbus 21 or 22 or 23 or 24. At best, a longtime Nimbus buyer is probably comparing a brand new pair of e.g. Nimbus 24 against their degraded but broken-in Nimbus 23 and their memory of how the Nimbus 23 felt when new. (And their body is a year or two older at that point..)
- Because it's such a long-running line of shoes, there could certainly be year-to-year variations... but it's hard to imagine there was an actual 5 or 10 or 25 year downward slope. I mean, otherwise at that point the shoes would just be instantly injuring you or falling apart in a week
- Also because it's such a long-running product line and (aside from bleeding-edge professional marathon/track shoes) sneaker manufacturing in general kind of seems like a solved problem... it seems like all of the possible cost optimizations have already been optimized. I don't really think there's much of a manufacturing or bill-of-materials cost difference between $20 running shoes and $200 running shoes anyway -- I'd be pretty surprised if "quality shrinkflation" was really much of a viable way for ASICS to save a few pennies.
jug · · focus · HN ↗
<a href="https://www.bridgebench.ai/nerf-bench" rel="nofollow">https://www.bridgebench.ai/nerf-bench
They test it on launch day, then benchmark it against that. A deviation of above 10% is considered a change. They're currently tracking Opus 5.5 and GPT-6 Astra.
This bench famously detected a degradation of Opus 4.6 which Anthropic later blogged about. I personally think people sense nerfs more often than they happen and that it's often about honeymoon effects.
nsarrazin · · focus · HN ↗
It doesn’t mean the big labs don’t also nerf models! But if they didn’t you’d still have users complaining.
booty · · focus · HN ↗
I was like, wow, I guess the shoes must be literal torture devices full of MRSA-covered broken glass at this point. They've been getting continuously worse for 24 consecutive years!
Of course, what was really happening is that they were not getting worse, but naturally every year there was some small percentage of vocal dissatisfied users, while the silent majority simply enjoyed their shoes and didn't have much to say about them.
(The sorta-opposite happens in sneaker reviews as well. People will gush about how cushy the sole in some particular new sneaker is. Well, yeah, of course it's cushy -- you're comparing a new sneaker to your old sneaker where the foam had lost its bounce...)
MikeTheGreat · · focus · HN ↗
Shrinkflation is a thing, which people suddenly started noticing in the past 5 years.
There's also "the Schlitz Mistake", which I've heard summarized as "most customers won't notice if you take your product's quality from A to B (or C), but they definitely will if you take it from A to J (or A to M)"
At this point I kinda assume that any company releasing year updates to a physical product that _doesn't_ take the opportunity to trim costs / reduce quality would be vulnerable to a shareholder lawsuit for leaving money on the table...
aesthesia · · focus · HN ↗
MikeTheGreat · · focus · HN ↗
and
On the other hand that idea that "companies exist to make money for shareholders, to ONLY make money for shareholders, and doing any other than maximizing shareholder returns is bad" is pretty commonly accepted (and, I believe, enshrined in US law)
skinfaxi · · focus · HN ↗
michaelmrose · · focus · HN ↗
[dead]
ehe78qhe · · focus · HN ↗
It's been much longer than that: <a href="https://en.wikipedia.org/wiki/Toblerone#2016_size_changes" rel="nofollow">https://en.wikipedia.org/wiki/Toblerone#2016_size_changes
xnorswap · · focus · HN ↗
<a href="https://www.bbc.co.uk/news/uk-44910195" rel="nofollow">https://www.bbc.co.uk/news/uk-44910195
Visually it looks like they took out half the peaks.
Your mind tells you that for every gap there used to be a peak there, regardless of the truth.
m10i · · focus · HN ↗
Summarizes the video game industry pretty well
pnt12 · · focus · HN ↗
MikeTheGreat · · focus · HN ↗
booty · · focus · HN ↗
That's certainly common!
I think ASICS' running shoes were a fun example of where this was probably not the case.
- I certainly didn't notice a difference in that time, though I'm admittedly not much of an actual runner
- Serious runners might notice small technical differences, but the negative user reviews didn't seem to indicate those were the people making the complaints
- The overall user reviews remained positive
- The competition in the shoe market is incredibly fierce; I'm not sure a brand could tank their quality and survive for long
- Let's not forget the other big variable: the wearers' bodies, particularly their feet. Now, those definitely do change over time -- most often for the worse, sadly!
- I doubt anybody was blind A/Bing a pair of Nimbus 20 against a pair of Nimbus 21 or 22 or 23 or 24. At best, a longtime Nimbus buyer is probably comparing a brand new pair of e.g. Nimbus 24 against their degraded but broken-in Nimbus 23 and their memory of how the Nimbus 23 felt when new. (And their body is a year or two older at that point..)
- Because it's such a long-running line of shoes, there could certainly be year-to-year variations... but it's hard to imagine there was an actual 5 or 10 or 25 year downward slope. I mean, otherwise at that point the shoes would just be instantly injuring you or falling apart in a week
- Also because it's such a long-running product line and (aside from bleeding-edge professional marathon/track shoes) sneaker manufacturing in general kind of seems like a solved problem... it seems like all of the possible cost optimizations have already been optimized. I don't really think there's much of a manufacturing or bill-of-materials cost difference between $20 running shoes and $200 running shoes anyway -- I'd be pretty surprised if "quality shrinkflation" was really much of a viable way for ASICS to save a few pennies.