> Of course, the more you know about a subject, the less convincing the AI's responses are.
This is said all the time by AI skeptics and I think it's right in some areas and massively wrong in others.
I know (or at least assume I know) a lot about certain coding domains where frontier models also show convincing ability. And we know that frontier LLMs really do excel in some areas of mathematics (i.e. when an inexpert human was able to prompt the models to derive a closer bound on the Riemann Hypothesis).
OTOH I know those same models struggle to do things I'm not an expert in (e.g. writing English in a captivating way) because I read their output and have taste.
I think part of this comes from the fact that LLMs are surprisingly good at logic but roughly about as good as expected on information accuracy.
LLMs are not convincing to me in the domain I did grad school...but neither is Wikipedia, or Reddit, or random pop sci books. And LLMs are basically just summarizing those things.
But when made to work through difficult arbitrary logic (like coding), they are very impressive.
I think this also explains why people gripe a lot about LLM coding _style_, but concede that LLMs do totally fine on coding _correctness_ in 2026
This sounds plausible. And it's also very fixable!
These days people don't interact with raw LLMs: they interact with systems and harnesses that deal with chain-of-though and tool calls etc.
I don't think we can honestly expect an LLM's weights to encode a large amount of information accurately. But we can expect the whole system that you interact with that includes the LLM to be able to cite its sources and go digging etc.
So the LLM-system can become as accurate as our best sources.
Of course, figuring out how to get the maximum of information from the sources available is a big deal. See eg how many economists or epidemiologists can build entire careers out of noticing 'natural experiments', ie figuring how to use data that 'nature' created and that might already be collected to answer interesting questions about causal relationships.
> I think this also explains why people gripe a lot about LLM coding _style_, but concede that LLMs do totally fine on coding _correctness_ in 2026
I actually have gripes about correctness, too. But I suspect here the answer is also: more proving, more automated test generation (like fuzzing and property based testing etc), more formal methods.
As a really simple and somewhat silly example: I have much better results getting AI agents to write good Rust code, than I have with Python. A good part of that is that for Rust I can ask the agent to make both the compiler and clippy::pedantic happy. That gives a lot of good feedback, that I didn't have to engineer myself.
More perhaps at the appearance of logic. I still find all models make easy to find mistakes in logic, if you think carefully about what they say. Usually they allow "close enough" assumptions that are not, actually, close enough.
When I ask about acoustics, they still often make incorrect assumptions, e.g. overlooking that they are speaking of logarithmic display of digital levels when the discussion has shifted to SPL (sound pressure level in air).
The more you know about a subject the better you can prompt AI, steer it toward the correct path, and recognize when it hallucinates or strays. Current generation AI is an automated memory-enhancement and thinking-accelerator tool, not a substitute for understanding or something that eliminates the need to think. A "mech suit for your brain" is the best analogy I've heard.
This is why good programmers get better results when vibe coding than non-programmers or poor programmers.
Aren't the recent results in mathematics actually stronger evidence for his point? Although the models may be capable of generating proofs they aren't coming out with the same level of quality of a human discovered and communicated proof. Providing a gobbledy-gook yet technically correct proof (generated at least in part by brute force) lacks the qualities of an expert produced proof because they fail to communicate insight or understanding about why the theorem is true.
Gaining and successfully communicating insight and understanding from a proof you discovered is additional work that human mathematicians do. It's not just some side-product of proof-finding (at least not to the degree usually needed to publish). That AI models don't provide this is mostly proof that the model wasn't asked to do this work. Either because the prompter didn't know or didn't care
But there are also plenty of examples of humans providing technically correct proofs without any elaboration. Usually they get ignored, unless they are famous or the problem they solved was famous
It seems to me that often experts from some field will think less of other experts, basically because they have built a different understanding framework. So they both may be equally competent but perceive the other as less competent, and that is just based on the material, excluding some ego stuff.
> (e.g. writing English in a captivating way) because I read their output and have taste.
Concur. In addition to taste, we also have a point of view, a unique voice (nobody loves corporate- or group-speak), and can iterate on our message as we deliver it to an ever wider circle of people.
I can tell whether AI responses in subject matter I am (or was) not familiar with are correct when I can "make them work" for me, i.e. whether they actually solve my problem, and on the whole they have been pretty solid.
Coding of course is something I know well and it obviously works very well for me there, but computer vision was not, and it has helped me solve lots of useful, bespoke problems because I can very literally "see" if it works.
However, it has even helped me solve problems in arbitrary matters far outside my expertise. Choice example: a complicated multi-airline, multi-jurisdiction flight delay compensation case which companies like AirHelp turned away, and ChatGPT got me literally hundreds of dollars that neither airline was willing to hand out. It told me what to say to whom and why, and when I said it, the responsible airline capitulated.
What could be more convincing than cold, hard $$$?
The trick of course, is to figure out how to validate the output of the AI, which can take some effort on our part. But many would rather downplay the technology than take an honest crack at making it work for them, and I suspect it's because they're already prejudiced against it or their incentives are otherwise not aligned.
muglug · · focus · HN ↗
This is said all the time by AI skeptics and I think it's right in some areas and massively wrong in others.
I know (or at least assume I know) a lot about certain coding domains where frontier models also show convincing ability. And we know that frontier LLMs really do excel in some areas of mathematics (i.e. when an inexpert human was able to prompt the models to derive a closer bound on the Riemann Hypothesis).
OTOH I know those same models struggle to do things I'm not an expert in (e.g. writing English in a captivating way) because I read their output and have taste.
howunfortunate · · focus · HN ↗
LLMs are not convincing to me in the domain I did grad school...but neither is Wikipedia, or Reddit, or random pop sci books. And LLMs are basically just summarizing those things.
But when made to work through difficult arbitrary logic (like coding), they are very impressive.
I think this also explains why people gripe a lot about LLM coding _style_, but concede that LLMs do totally fine on coding _correctness_ in 2026
eru · · focus · HN ↗
These days people don't interact with raw LLMs: they interact with systems and harnesses that deal with chain-of-though and tool calls etc.
I don't think we can honestly expect an LLM's weights to encode a large amount of information accurately. But we can expect the whole system that you interact with that includes the LLM to be able to cite its sources and go digging etc.
So the LLM-system can become as accurate as our best sources.
Of course, figuring out how to get the maximum of information from the sources available is a big deal. See eg how many economists or epidemiologists can build entire careers out of noticing 'natural experiments', ie figuring how to use data that 'nature' created and that might already be collected to answer interesting questions about causal relationships.
> I think this also explains why people gripe a lot about LLM coding _style_, but concede that LLMs do totally fine on coding _correctness_ in 2026
I actually have gripes about correctness, too. But I suspect here the answer is also: more proving, more automated test generation (like fuzzing and property based testing etc), more formal methods.
As a really simple and somewhat silly example: I have much better results getting AI agents to write good Rust code, than I have with Python. A good part of that is that for Rust I can ask the agent to make both the compiler and clippy::pedantic happy. That gives a lot of good feedback, that I didn't have to engineer myself.
netdevphoenix · · focus · HN ↗
eru · · focus · HN ↗
agumonkey · · focus · HN ↗
QuantumGood · · focus · HN ↗
When I ask about acoustics, they still often make incorrect assumptions, e.g. overlooking that they are speaking of logarithmic display of digital levels when the discussion has shifted to SPL (sound pressure level in air).
the__alchemist · · focus · HN ↗
api · · focus · HN ↗
This is why good programmers get better results when vibe coding than non-programmers or poor programmers.
doesnotexist · · focus · HN ↗
wongarsu · · focus · HN ↗
But there are also plenty of examples of humans providing technically correct proofs without any elaboration. Usually they get ignored, unless they are famous or the problem they solved was famous
comboy · · focus · HN ↗
andsoitis · · focus · HN ↗
Concur. In addition to taste, we also have a point of view, a unique voice (nobody loves corporate- or group-speak), and can iterate on our message as we deliver it to an ever wider circle of people.
keeda · · focus · HN ↗
Coding of course is something I know well and it obviously works very well for me there, but computer vision was not, and it has helped me solve lots of useful, bespoke problems because I can very literally "see" if it works.
However, it has even helped me solve problems in arbitrary matters far outside my expertise. Choice example: a complicated multi-airline, multi-jurisdiction flight delay compensation case which companies like AirHelp turned away, and ChatGPT got me literally hundreds of dollars that neither airline was willing to hand out. It told me what to say to whom and why, and when I said it, the responsible airline capitulated.
What could be more convincing than cold, hard $$$?
The trick of course, is to figure out how to validate the output of the AI, which can take some effort on our part. But many would rather downplay the technology than take an honest crack at making it work for them, and I suspect it's because they're already prejudiced against it or their incentives are otherwise not aligned.