3.8 Max is the most “grounded” model I think - talks generally normal, doesn’t go crazy and start doing things (I see you Gemini), has good design choices and isn’t overly nitpicky. But god it’s slow. And only available from Alibaba. Their token plan is stingy too. If I had to pick the “old reliable boring” LLM, a modern Claude 4.5 if you will, Qwen is my choice. Hopefully they don’t RL it to oblivion.
It’s absolutely down to their post-training RL, yeah. It’s where most of its strongest behaviour comes from, with regards to this kind of agentic behaviour
Others have given examples, but here's the theory: <a href="https://www.lesswrong.com/posts/fuSaKr6t6Zuh6GKaQ/when-is-goodhart-catastrophic" rel="nofollow">https://www.lesswrong.com/posts/fuSaKr6t6Zuh6GKaQ/when-is-go...
Reinforcement Learning (in LLMs) trains via gradient descent on a reward signal that's an imperfect proxy for the actual goal of the engineers doing the training. So, under mild optimization pressure, you get increasingly more of what you want, because that's the easiest way to increase the metric.
But as the optimization pressure increases, so do the ways to increase the metric by doing increasingly weird things. If the full action space grows sufficiently faster than the "things you actually want" subset, the amount of "things you actually want" goes to 0 under sufficient RL.
I think we are starting be on that territory that regular software development is suffering, current models are great for benchmarks and one-shots but in daily development models are too eager and try to force patterns like excessive tests in every turn.
conception · · focus · HN ↗
rubslopes · · focus · HN ↗
What would that mean in this context?
cleaning · · focus · HN ↗
smallerfish · · focus · HN ↗
pennomi · · focus · HN ↗
I swear I spend more time telling Claude not to do things than telling it what to do.
mdp2021 · · focus · HN ↗
But is that because of training, or can that be (also? mostly?) an effect of the "system prompt"?
girvo · · focus · HN ↗
vintermann · · focus · HN ↗
disgruntledphd2 · · focus · HN ↗
Personally, I think this is a bad idea, but someone's gotta build the Machine God I guess.
khafra · · focus · HN ↗
Reinforcement Learning (in LLMs) trains via gradient descent on a reward signal that's an imperfect proxy for the actual goal of the engineers doing the training. So, under mild optimization pressure, you get increasingly more of what you want, because that's the easiest way to increase the metric.
But as the optimization pressure increases, so do the ways to increase the metric by doing increasingly weird things. If the full action space grows sufficiently faster than the "things you actually want" subset, the amount of "things you actually want" goes to 0 under sufficient RL.
conception · · focus · HN ↗
antupis · · focus · HN ↗