> Before approving construction, I would want communities of humans to understand why the design works and what justifies confidence in its safety. I would hope that we all would.
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
Very few people scrutinise assembly in 2026 as compiler generated code is 'good enough'. LLMs are beginning to do the same with higher level languages.
Without bashing anyone in particular, a certain OS-vendor's desktop apps, have been 'good enough' to ship, but with p*ss-poor performance in many cases for the last decade or so. We crossed the 'good enough' Rubicon a few years back in terms of what end users receive as a finished app.
Hopefully LLMs will eventually bridge that last gap of efficiency when generating higher-level code that not only works, but is efficient. Maybe there's a future where they generate the final binary without even invoking a compiler.
but there is a difference between deterministic compilation and non-deterministic LLM code. Of course I don't think this is an issue for toy problems and simple codebases, but for non-trivial problems I think it will be an issue.
When I compile C code I know that maybe it will not be as efficient as it could be if I had written it in Assembly, but there will be a biunivocal correspondence between C and Assembly. If instead I use an LLM to rewrite a feature of a codebase I can't be sure that it still functions like the original one.
I acknowledge that this is an issue with human programmers too, but I don't see a clear way forward, even if I'm really interested in LLM compilers being a thing. Maybe we will use them for non important code, and we will keep writing system critical stuff by hand.
> but there is a difference between deterministic compilation and non-deterministic LLM code
But not the point of my comment. Computer programming has been a progression of physically wiring up valves, to soldering transistors, to punched cards, assembly, then higher level languages. Now we have natural language models.
The analogy being each that most people don't care about the assembly generated as the code works and it's really performant/efficient. Humans can still optimise assembly, but there's vanishingly small marginal gains for all but the most intensive/low-level tasks.
If LLMs produce things that work, and are indistinguishable from a careful human programmer (i.e. with some level of acceptable performance), people will simply stop looking at the high level code as the end result works, in the same way most people stopped looking at generated assembly after 8 bit computers (for example, as most games were written in raw assembly for... perforamance), as it was good enough.
> If instead I use an LLM to rewrite a feature of a codebase I can't be sure that it still functions like the original one.
Right now, with existing static analysis tooling, you can ask it to write a full suite of unit tests capturing existing behaviour without modifying the existing code with 100% code coverage, and start there. Plus fuzz tests as well. I actually have marginally more confidence in that than a human being doing it.
pyridines · · focus · HN ↗
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
DrBazza · · focus · HN ↗
Without bashing anyone in particular, a certain OS-vendor's desktop apps, have been 'good enough' to ship, but with p*ss-poor performance in many cases for the last decade or so. We crossed the 'good enough' Rubicon a few years back in terms of what end users receive as a finished app.
Hopefully LLMs will eventually bridge that last gap of efficiency when generating higher-level code that not only works, but is efficient. Maybe there's a future where they generate the final binary without even invoking a compiler.
gekoxyz · · focus · HN ↗
DrBazza · · focus · HN ↗
But not the point of my comment. Computer programming has been a progression of physically wiring up valves, to soldering transistors, to punched cards, assembly, then higher level languages. Now we have natural language models.
The analogy being each that most people don't care about the assembly generated as the code works and it's really performant/efficient. Humans can still optimise assembly, but there's vanishingly small marginal gains for all but the most intensive/low-level tasks.
If LLMs produce things that work, and are indistinguishable from a careful human programmer (i.e. with some level of acceptable performance), people will simply stop looking at the high level code as the end result works, in the same way most people stopped looking at generated assembly after 8 bit computers (for example, as most games were written in raw assembly for... perforamance), as it was good enough.
> If instead I use an LLM to rewrite a feature of a codebase I can't be sure that it still functions like the original one.
Right now, with existing static analysis tooling, you can ask it to write a full suite of unit tests capturing existing behaviour without modifying the existing code with 100% code coverage, and start there. Plus fuzz tests as well. I actually have marginally more confidence in that than a human being doing it.