Most managers don't read the code of their direct reports. They just trust that the code is "good enough" and that there are enough other processes in place to catch bugs before they cause too much damage. I think I'm a pretty skilled programmer, or I'm at least good enough at interviewing to convince other people of that, but it's pretty obvious from comparing LLM generated code to code I'd write myself that in many domains LLMs are superior to me. They do make mistakes, but so do I, and those mistakes are eventually found and corrected.
We're in the very early stages of LLM driven programming, so it's hard to say how it will all shake out, but my anecdotal experience is that LLM written software is very reliable and easy to extend and develop. I have a side project that is about 90% LLM generated code (about 25k lines of production code and a similar amount of test code). This is a revenue generating product and I've had no issues with reliability, security, or performance.
For what it's worth this app is a rails app and I have no plans to switch to anything else. Rails works nicely, the LLMs extend it easily, and almost everything is I/O bound so I don't need C++/Rust level performance.
> I think I'm a pretty skilled programmer, or I'm at least good enough at interviewing to convince other people of that, but it's pretty obvious from comparing LLM generated code to code I'd write myself that in many domains LLMs are superior to me.
there's an important caveat here - LLMs absolutely can write better code than me, but they can also write far worse code, and often don't seem to be able to tell the difference. I find myself spending a lot of time prompting the bots to refine their code in specific ways that I only know about because I read the generated output.
but my anecdotal experience is that LLM written
software is very reliable and easy to extend and
develop.
Extremely refreshing to read. The whole thing, not just this statement.
Too much discourse creates a false dichotomy of "awesome, wonderful, hand-crafted code" vs "shitty AI slop."
A lot of hand-crafted code is pretty bad. This is true even with talented engineers: there are often edge cases they did not think of... and in many cases, could not have thought of.
And AI code is pretty good if it's steered and vetted by a knowledgeable engineer. It is clear to me, and has been clear for a long time, that "talented engineer plus AI" is the winning combination and will remain that way for at least the near future.
I still review PRs, all of which are written by agents. I almost never have meaningful feedback to offer unless it's feedback on the architectural approach. And even then, a lot of architectural/coding patterns that I've been a stickler for in the past mean less to me now because those patterns served to create maintainable code for human beings, which just isn't a priority anymore.
At our company (big Rails/React monolith) all PRs do get a human review. Now, I use LLMs to accelerate my reviews, but there's still a lot of space for humans.
- LLMs struggle, or at least burn oodles of tokens, on big complex codebases. Managing tech debt helps the agents as well IMO.
- If you ask an LLM to review a non-trivial pull request, it nearly always seems to find at least 10 issues. Many of which are not worth pursuing. Often, they would introduce a lot of extra complexity to harden the code against scenarios we don't care about or can never happen. So there's a lot of room for human judgement there. Typical scenario - the LLM reviewer finds 10 issues and I judge that 2-3 are worth pursuing
- As you said, often the architectural approach sucks even with frontier LLMs. Usually from a lack of problem space / usage scenarios more than a lack of technical chops
- LLMs tend to err on the side of overengineering the shit out of everything
I do not see how LLMs will be able to manage huge, hairy polygot codebases without humans-in-the-loop in the near future.
Given the pace of progress, I'd be a fool to bet against LLMs in the medium term future. But I think it's far from a given. LLMs themselves, and large codebases, are essentially many-to-many problems approaching something like O(n^n).
Maybe. The market will figure out if this is viable, I guess.
For smaller projects or short-lived projects, "AI without talented engineer" is a viable choice already.
But a lot of projects are big and long lived.
The difference in complexity between 1K lines of code and 10K lines of code isn't merely 10x. It's usually more like 100x.
So even if AI becomes 10x better than today (which seems reasonable) there's going to be a level of complexity beyond which you still need some kind of talented engineer-type(s) steering the AI.
*"talented engineer plus AI" is the winning combination and will remain that way for at least the near future.
Talented programmers will retire and Father Time takes care of the rest. Where will the new wave of talented programmers come from? I use AI, but I am a hobbyist.
jeffreyrogers · · focus · HN ↗
We're in the very early stages of LLM driven programming, so it's hard to say how it will all shake out, but my anecdotal experience is that LLM written software is very reliable and easy to extend and develop. I have a side project that is about 90% LLM generated code (about 25k lines of production code and a similar amount of test code). This is a revenue generating product and I've had no issues with reliability, security, or performance.
For what it's worth this app is a rails app and I have no plans to switch to anything else. Rails works nicely, the LLMs extend it easily, and almost everything is I/O bound so I don't need C++/Rust level performance.
zem · · focus · HN ↗
there's an important caveat here - LLMs absolutely can write better code than me, but they can also write far worse code, and often don't seem to be able to tell the difference. I find myself spending a lot of time prompting the bots to refine their code in specific ways that I only know about because I read the generated output.
booty · · focus · HN ↗
Too much discourse creates a false dichotomy of "awesome, wonderful, hand-crafted code" vs "shitty AI slop."
A lot of hand-crafted code is pretty bad. This is true even with talented engineers: there are often edge cases they did not think of... and in many cases, could not have thought of.
And AI code is pretty good if it's steered and vetted by a knowledgeable engineer. It is clear to me, and has been clear for a long time, that "talented engineer plus AI" is the winning combination and will remain that way for at least the near future.
ncphillips · · focus · HN ↗
prescriptivist · · focus · HN ↗
booty · · focus · HN ↗
- LLMs struggle, or at least burn oodles of tokens, on big complex codebases. Managing tech debt helps the agents as well IMO. - If you ask an LLM to review a non-trivial pull request, it nearly always seems to find at least 10 issues. Many of which are not worth pursuing. Often, they would introduce a lot of extra complexity to harden the code against scenarios we don't care about or can never happen. So there's a lot of room for human judgement there. Typical scenario - the LLM reviewer finds 10 issues and I judge that 2-3 are worth pursuing - As you said, often the architectural approach sucks even with frontier LLMs. Usually from a lack of problem space / usage scenarios more than a lack of technical chops - LLMs tend to err on the side of overengineering the shit out of everything
I do not see how LLMs will be able to manage huge, hairy polygot codebases without humans-in-the-loop in the near future.
Given the pace of progress, I'd be a fool to bet against LLMs in the medium term future. But I think it's far from a given. LLMs themselves, and large codebases, are essentially many-to-many problems approaching something like O(n^n).
pjmlp · · focus · HN ↗
booty · · focus · HN ↗
For smaller projects or short-lived projects, "AI without talented engineer" is a viable choice already.
But a lot of projects are big and long lived.
The difference in complexity between 1K lines of code and 10K lines of code isn't merely 10x. It's usually more like 100x.
So even if AI becomes 10x better than today (which seems reasonable) there's going to be a level of complexity beyond which you still need some kind of talented engineer-type(s) steering the AI.
stasomatic · · focus · HN ↗
Talented programmers will retire and Father Time takes care of the rest. Where will the new wave of talented programmers come from? I use AI, but I am a hobbyist.