Writing is fundamentally the transfer of information from your brain to my brain. If you have 1000 bits of semantic information you want to transfer, you can't give 300 bits of semantic information to an LLM and have it fill in the remaining 700, because it doesn't know what those 700 bits are. If it's able to guess those 700 bits correctly, then they aren't true semantic information, and you really only have 300 bits you want to transfer. You might as well transfer those bits to me directly, rather than having the LLM add on an extra superfluous 700 bits that I then have to filter out.
I’m not sure the 300-bit → 1,000-bit framing applies in all instances. The 300 bits may be a compressed cue to a much fuller idea. The AI can combine that cue with its prior knowledge to help reconstruct what the prompter was trying to express, with the prompter then verifying whether it’s right. Without the relevant prior knowledge for reconstruction, or the prompter for verification, it becomes much harder to know whether you’ve reconstructed the intended idea.
Unless that 700 bit was transferred on a separate occasion the inferred 700 bits is not true information, anyone could have reconstructed it from the 300 bits.
That assumes each bit is a coinflip, doesn't it?
Even Markov chain autocorrect tools do better than 50% odds*, and even GPT-2 was significantly better than that kind of autocorrect.
* at the word level; IDK how redundant/efficient language is when it comes to bits-worth-of-fact-claims-per-word. But "your cat is sitting on my" -> [mat, laundry, roof, head, belly, laptop, microwave, …] clearly has many bits of information, and a Markov chain will encode the most likely next word even if the user doesn't know what the most likely next word is. Verifying where the cat is sitting is also very easy, as is correction.
Far more than 1 bit for most attempts: they need that just to be able to write coherent sentences, and a lot more to be coherent sentences on the right topic.
Some specific conclusions would be far less than 1 bit.
The average will depend on both the question and the AI.
LLM is not a random symbol generator (hint: training data is not random), and no reasonable person is going to just prompt an LLM and send its output without giving it at least cursory check (at the very least so that blatantly stupid hallucinations don't paint the sender as inconsiderate or incompetent).
That check alone can add bits to the final signal.
> and no reasonable person is going to just prompt an LLM and send its output without giving it at least cursory check
I'm reminded of an old quote:
The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.
I'm not sure we're working in the same framing here. For one LLMs are random, sure you can fix the seed but you don't have to, you can even replace the RNG with true random noise.
So there are 2^300 possible ideas, only 2 outcomes from the cursory check, how do you get 2^700 outcomes? Most of those are just random variations the LLM added which is not a transfer of information. You would be lucky to even identify which of the 2^300 ideas was being conferred.
"Random" is too loose a word, but they were responding in a context where it meant coin flips.
LLMs are not even odds on all possible outputs, they are biased towards patterns which are upvoted by the training mechanism (at a minimum: the source material, RLHF, and synthetic data).
The information any trained model transfers to output, is information it gained during its training.
No single human is capable of having consumed all that training data.
There comes a point where someone isn't so much talking to the sender as having an unsolicited AI chat. Especially when most of the information didn't come from the sender in the first place.
It's actually hard to define the difference between information that comes from the model and just random variation, maybe something to do with the cross entropy between the sender and the model?
> There comes a point where someone isn't so much talking to the sender as having an unsolicited AI chat. Especially when most of the information didn't come from the sender in the first place.
Indeed.
The best case is a P vs NP situation: can the claims from the AI be easily verified, or not?
This does not excuse people too lazy (or overly impressed*) who fail to attempt the verification.
> It's actually hard to define the difference between information that comes from the model and just random variation, maybe something to do with the cross entropy between the sender and the model?
Mm.
Thanks to a philosophy course I did half a lifetime ago, I think there's a fundamental problem defining "information" in this context. It feels like it should mean "knowledge" because the discussions about Shannon entropy and transmission channels assumes there is an actual source-of-truth, but my conclusion from discussions about why "knowledge" can't just mean a "justified true belief" is thay I now don't believe we can do better than "belief"; an LLM can generate tokens that change your beliefs, but ultimately neither you nor I nor some annoying colleage who has made themselves redundant to the LLM, can be an oracle with definitely-true knowledge.
(I have of course tried asking an LLM about this thread; I don't feel it illuminated anything new for me, none of what it suggested made it into this comment).
* In the early days of LLMs, I was overly-impressed. Then I realised we were doing the same thing with LLMs today that we did with 3D graphics in the 90s, where every new engine was hailed as "photorealistic" only to be dismissed 6 months later when something better came along: <a href="https://archive.org/details/nextgen-issue-26" rel="nofollow">https://archive.org/details/nextgen-issue-26
Only now it's every 11 weeks rather than 6 months.
hatthew · · focus · HN ↗
Writing is fundamentally the transfer of information from your brain to my brain. If you have 1000 bits of semantic information you want to transfer, you can't give 300 bits of semantic information to an LLM and have it fill in the remaining 700, because it doesn't know what those 700 bits are. If it's able to guess those 700 bits correctly, then they aren't true semantic information, and you really only have 300 bits you want to transfer. You might as well transfer those bits to me directly, rather than having the LLM add on an extra superfluous 700 bits that I then have to filter out.
zefalt · · focus · HN ↗
shiandow · · focus · HN ↗
TeMPOraL · · focus · HN ↗
shiandow · · focus · HN ↗
ben_w · · focus · HN ↗
Even Markov chain autocorrect tools do better than 50% odds*, and even GPT-2 was significantly better than that kind of autocorrect.
* at the word level; IDK how redundant/efficient language is when it comes to bits-worth-of-fact-claims-per-word. But "your cat is sitting on my" -> [mat, laundry, roof, head, belly, laptop, microwave, …] clearly has many bits of information, and a Markov chain will encode the most likely next word even if the user doesn't know what the most likely next word is. Verifying where the cat is sitting is also very easy, as is correction.
shiandow · · focus · HN ↗
ben_w · · focus · HN ↗
Some specific conclusions would be far less than 1 bit.
The average will depend on both the question and the AI.
shiandow · · focus · HN ↗
A LLM adds noise, not information. At least in this framing.
TeMPOraL · · focus · HN ↗
LLM is not a random symbol generator (hint: training data is not random), and no reasonable person is going to just prompt an LLM and send its output without giving it at least cursory check (at the very least so that blatantly stupid hallucinations don't paint the sender as inconsiderate or incompetent).
That check alone can add bits to the final signal.
ben_w · · focus · HN ↗
I'm reminded of an old quote:
TeMPOraL · · focus · HN ↗
shiandow · · focus · HN ↗
So there are 2^300 possible ideas, only 2 outcomes from the cursory check, how do you get 2^700 outcomes? Most of those are just random variations the LLM added which is not a transfer of information. You would be lucky to even identify which of the 2^300 ideas was being conferred.
ben_w · · focus · HN ↗
LLMs are not even odds on all possible outputs, they are biased towards patterns which are upvoted by the training mechanism (at a minimum: the source material, RLHF, and synthetic data).
The information any trained model transfers to output, is information it gained during its training.
No single human is capable of having consumed all that training data.
shiandow · · focus · HN ↗
It's actually hard to define the difference between information that comes from the model and just random variation, maybe something to do with the cross entropy between the sender and the model?
ben_w · · focus · HN ↗
Indeed.
The best case is a P vs NP situation: can the claims from the AI be easily verified, or not?
This does not excuse people too lazy (or overly impressed*) who fail to attempt the verification.
> It's actually hard to define the difference between information that comes from the model and just random variation, maybe something to do with the cross entropy between the sender and the model?
Mm.
Thanks to a philosophy course I did half a lifetime ago, I think there's a fundamental problem defining "information" in this context. It feels like it should mean "knowledge" because the discussions about Shannon entropy and transmission channels assumes there is an actual source-of-truth, but my conclusion from discussions about why "knowledge" can't just mean a "justified true belief" is thay I now don't believe we can do better than "belief"; an LLM can generate tokens that change your beliefs, but ultimately neither you nor I nor some annoying colleage who has made themselves redundant to the LLM, can be an oracle with definitely-true knowledge.
(I have of course tried asking an LLM about this thread; I don't feel it illuminated anything new for me, none of what it suggested made it into this comment).
* In the early days of LLMs, I was overly-impressed. Then I realised we were doing the same thing with LLMs today that we did with 3D graphics in the 90s, where every new engine was hailed as "photorealistic" only to be dismissed 6 months later when something better came along: <a href="https://archive.org/details/nextgen-issue-26" rel="nofollow">https://archive.org/details/nextgen-issue-26
Only now it's every 11 weeks rather than 6 months.