From TFA: "Eric Schwitzgebel writes that . . . There’s a huge cognitive difference between nodding along while reading something and actually productively generating a text. Two reasons: First, once the text is on the page, it’s easy to passively let the approximate word suffice, rather than thinking about word choice in the same effortful, active way we do when generating prose de novo. Second, as I suggested above, I doubt that human beings, even experts, have a good sense of all the factors that shape word choice -- everything they’re being sensitive to. You would have phrased it slightly differently, and even if you don’t know that, or why, a different signal is sent and received."
This is the best articulation I've seen of why simply reviewing and copy-editing does not provide remotely the same value as writing from scratch. I spent a considerable amount of time over the past two weeks reviewing and improving a work document that was the output of an LLM. Given the number of people involved and the final level of effort, I'm firmly convinced that writing it manually would have been faster and resulted in a higher quality product. Getting the wording right matters.
I really did like the author's framing there, but I think there is a different, simpler way to put it:
Why would you think that asking someone else (that is, another human) to write something (and then reviewing it) is the same thing as writing it yourself?
You may trust the other writer's opinions and knowledge, but it will not have the same tone, structure, word choice, understanding, or narrative flow as it would if you were to write it yourself.
And when it's an LLM, you should not trust it's "opinions" and "knowledge", because it does not have either of those things. The appearance of those things is just that, an appearance.
The key difference is that while an encyclopedia holds facts themselves, LLMs trained on that source material encode something more like a highly probable facsimile of those facts - the original fact was lost, LLMs are lossy, but can often be generated again with a decent level of accuracy by churning through stats about words, concepts, and relationships between them.
The whole catch is that they can often be regenerated. But LLMs (on their own, in their parametric memory - which is the result of training) don't have any conception of whether what they've generated is a real reproduction of some training material or whether they've invented something false that seemed probable based on their encoded stats. When the probability produces something contrary to what was in the training material, you get hallucinations.
They're very, very good predictive text models and can be very, very powerful when hooked up to other tools or outside databases. But its fundamentally lossy technology and all the books having been fed in doesn't guarantee all of the knowledge from those books can be spat back out.
Humans can learn texts by heart, even lots of text (I have some expertise on that, having studied opera singing). They can reproduce these texts accurately, deterministically and repeatably. An LLM is a statistical machine. It does not know any text by heart and it is by pure chance that it sometimes reproduces existing texts verbatim.
Humans sometimes don't remember things 100%. LLMs can produce certain text and be run deterministically.
I don't really understand this side of the debate other than as a gotcha tbh.
If LLM use atrophies your brain and skills that's bad. If it has a repulsive writing style that's bad.
I'm not sure what the debate about whether an AI is a statistical parrot unlike humans accomplishes. Is relying 100% on a bad human speechwriter somehow better?
It comes up frequently when discussing whether LLMs actually have intelligence or qualify as AI, especially in discussions on the path to achieving AGI.
The primary complaint seems to be that LLMs are held to a higher standard than humans, though I don't particularly buy that line of reasoning.
patrickmay · · focus · HN ↗
This is the best articulation I've seen of why simply reviewing and copy-editing does not provide remotely the same value as writing from scratch. I spent a considerable amount of time over the past two weeks reviewing and improving a work document that was the output of an LLM. Given the number of people involved and the final level of effort, I'm firmly convinced that writing it manually would have been faster and resulted in a higher quality product. Getting the wording right matters.
kelnos · · focus · HN ↗
Why would you think that asking someone else (that is, another human) to write something (and then reviewing it) is the same thing as writing it yourself?
You may trust the other writer's opinions and knowledge, but it will not have the same tone, structure, word choice, understanding, or narrative flow as it would if you were to write it yourself.
And when it's an LLM, you should not trust it's "opinions" and "knowledge", because it does not have either of those things. The appearance of those things is just that, an appearance.
card_zero · · focus · HN ↗
sethhochberg · · focus · HN ↗
The whole catch is that they can often be regenerated. But LLMs (on their own, in their parametric memory - which is the result of training) don't have any conception of whether what they've generated is a real reproduction of some training material or whether they've invented something false that seemed probable based on their encoded stats. When the probability produces something contrary to what was in the training material, you get hallucinations.
They're very, very good predictive text models and can be very, very powerful when hooked up to other tools or outside databases. But its fundamentally lossy technology and all the books having been fed in doesn't guarantee all of the knowledge from those books can be spat back out.
zmgsabst · · focus · HN ↗
JanNash · · focus · HN ↗
kjshsh123 · · focus · HN ↗
I don't really understand this side of the debate other than as a gotcha tbh.
If LLM use atrophies your brain and skills that's bad. If it has a repulsive writing style that's bad.
I'm not sure what the debate about whether an AI is a statistical parrot unlike humans accomplishes. Is relying 100% on a bad human speechwriter somehow better?
zdragnar · · focus · HN ↗
The primary complaint seems to be that LLMs are held to a higher standard than humans, though I don't particularly buy that line of reasoning.