I’m pretty sure 99% of what people perceive is the old model training on the usage logs wherever they were stuck.
Step 1.
Model can’t do something challenging
Step 2.
You try a bunch and fail
Step 3.
Anthropic trains on your usage data.
Your current code base and current problem are now in domain
Step 4.
Model comes out and you’re shocked when it can tackle the thing you were stuck on
Step 4.
Codebase drifts significantly and you try new problems you thought were a similar level. Your code is less familiar and the problem doesn’t have a bunch of failure cases in the train set.
Feels of it being worse on similar problems
I don't think so, because so called nerfing manifests itself in ridiculous code quality, or even in failing to do a comprehensive code analysis which results in "actually there is a bug in the implementation i've just done because this flow has 5 steps and i didn't bother to review them all before confidently laying down my plan", and this can happen 4 times in a row during a session.
This is definitely not about dealing with the frontier of AI. I wasn't part of the nerfing chord, but Astra changed my mind. Quality got me to upgrade from Pro x5 to Pro x20 on launch day. A couple of days later was dumb af, horrendous code quality etc...
Something fishy, or at least unethical is going on. Not sure it impacts API users though.
nbardy · · focus · HN ↗
Step 1. Model can’t do something challenging Step 2. You try a bunch and fail Step 3. Anthropic trains on your usage data. Your current code base and current problem are now in domain Step 4. Model comes out and you’re shocked when it can tackle the thing you were stuck on Step 4. Codebase drifts significantly and you try new problems you thought were a similar level. Your code is less familiar and the problem doesn’t have a bunch of failure cases in the train set. Feels of it being worse on similar problems
rednb · · focus · HN ↗
This is definitely not about dealing with the frontier of AI. I wasn't part of the nerfing chord, but Astra changed my mind. Quality got me to upgrade from Pro x5 to Pro x20 on launch day. A couple of days later was dumb af, horrendous code quality etc...
Something fishy, or at least unethical is going on. Not sure it impacts API users though.