I'm actively watching understanding slip away from developers, code review getting paired down to no comment checkmarks, and codebases go to bloated messes that nobody can read. Axioms like engineers must understand and take responsibility for the code they ship are getting torn down, and the products coming out are reflecting conway's law, becoming impenetrably obtuse and always "so complex there are no obvious deficiencies" (as opposed to "so simple there are no obvious deficiencies" which used to be the aim).
The one thing plan mode helped is for the humans to get an understanding of the strategy, and be able to poke around and look at the design and architecture. You can achieve this with some self discipline and keeping shorter leashes on agents, but it feels like a losing battle. The best devs still put out good code, but the poor devs are learning nothing while their metrics look great. I can't help but think we are racking up immense amounts of debt that will very soon become due.
I see this complaint frequently about losing track of what the agents are doing, and I agree you do need to understand your system. But there seems to be this baked in assumption that if you lose track, you now need to manually wade through this massive mess to untangle it and maybe that is impossible. I don't agree.
If you don't understand the codebase, ask the agent to explain it to you. I'm not not kidding. Modern frontier models are fantastic as this - even more so than actually writing the code. It can tell you in words. It can generate architectural diagrams and sequence diagrams. It can write tests and scripts that prove it's assumptions. It can happily refactor so that the system design is aligned with your preferences.
Once you accept this, you can stop worrying so much about it and instead focusing on building the architectures and tools that lets the agents succeed better and faster - so called closed loops or agents prompting agents. Build systems that are more easily verifiable and deterministic so the agent can write very powerful property based tests. Focus more on what and why you are building, how to make sure all external properties are verifiable and leave the internals to the agents. The code is not really for us anymore.
My experience is this is great when the model surfaces something to you. But I'm constantly caught off-guard by things the model didn't volunteer, things I would have quickly stumbled upon if I was working on code the traditional way. The model didn't think it was relevant but I sure do.
In my experience having the agent explain the code doesn’t work very well for real world apps, even the parts written by humans. For example we tried using to generate diagrams, class hierarchies, etc as part of documentation. If you don’t know the code it looks great. If you do, it’s focusing on all the wrong things, missing the mental model, and ignoring lots of important bits. And Claude tends to be extremely verbose to the point of muddling things.
In my experience it's really good at making you feel that you understand things.
Then when you actually dig into the code, there are many things that are not like you'd expect.
When you've experienced that a few times, you stop trusting that the agent gives you the full picture - for good reason.
When I review AI generated code I generally find so many flaws that it makes it hard for me to believe that those who are not reviewing their output are not just fooling themselves. Maybe not all the time, but quite often.
One such recent example was an SSO simulator for a local env. Instead of using a cookie to remember who was logged in, the agent remembered the last log in a variable, assuming the the next requests would come from that login.
This snowballed into our tests, where later agents had created helper tools for working around the SSO simulators statefulness.
Things improve drastically however if you spin up a second session and ask it to adversarially review everything that the first session produces (this goes for everything: not just code, but also design, planning, and explanations).
This works even better if you use models from different families to do so.
I have, on a lark, reached 20 to 30 adversarial sessions a few times. For some tasks, the sessions will just never converge on anything that yet another session won't find fault with, recommending an alternative already ruled out by another session. Even if all the reasoning in between was documented, the new sessions will endlessly claim to find flaws in past reasoning.
taurath · · focus · HN ↗
The one thing plan mode helped is for the humans to get an understanding of the strategy, and be able to poke around and look at the design and architecture. You can achieve this with some self discipline and keeping shorter leashes on agents, but it feels like a losing battle. The best devs still put out good code, but the poor devs are learning nothing while their metrics look great. I can't help but think we are racking up immense amounts of debt that will very soon become due.
frankc · · focus · HN ↗
If you don't understand the codebase, ask the agent to explain it to you. I'm not not kidding. Modern frontier models are fantastic as this - even more so than actually writing the code. It can tell you in words. It can generate architectural diagrams and sequence diagrams. It can write tests and scripts that prove it's assumptions. It can happily refactor so that the system design is aligned with your preferences.
Once you accept this, you can stop worrying so much about it and instead focusing on building the architectures and tools that lets the agents succeed better and faster - so called closed loops or agents prompting agents. Build systems that are more easily verifiable and deterministic so the agent can write very powerful property based tests. Focus more on what and why you are building, how to make sure all external properties are verifiable and leave the internals to the agents. The code is not really for us anymore.
spiffytech · · focus · HN ↗
abbefaria27 · · focus · HN ↗
jeppester · · focus · HN ↗
Then when you actually dig into the code, there are many things that are not like you'd expect.
When you've experienced that a few times, you stop trusting that the agent gives you the full picture - for good reason.
When I review AI generated code I generally find so many flaws that it makes it hard for me to believe that those who are not reviewing their output are not just fooling themselves. Maybe not all the time, but quite often.
One such recent example was an SSO simulator for a local env. Instead of using a cookie to remember who was logged in, the agent remembered the last log in a variable, assuming the the next requests would come from that login.
This snowballed into our tests, where later agents had created helper tools for working around the SSO simulators statefulness.
int_19h · · focus · HN ↗
Things improve drastically however if you spin up a second session and ask it to adversarially review everything that the first session produces (this goes for everything: not just code, but also design, planning, and explanations).
This works even better if you use models from different families to do so.
sdevonoes · · focus · HN ↗
sdiupIGPWEfh · · focus · HN ↗
int_19h · · focus · HN ↗
Like with most other things of this nature, the returns are diminishing, but ultimately it's for you to decide how much is enough.
[deleted] · · focus · HN ↗
[deleted]
gantam · · focus · HN ↗
[dead]