Writing is fundamentally the transfer of information from your brain to my brain. If you have 1000 bits of semantic information you want to transfer, you can't give 300 bits of semantic information to an LLM and have it fill in the remaining 700, because it doesn't know what those 700 bits are. If it's able to guess those 700 bits correctly, then they aren't true semantic information, and you really only have 300 bits you want to transfer. You might as well transfer those bits to me directly, rather than having the LLM add on an extra superfluous 700 bits that I then have to filter out.
For a given information model, compression isn't a useful term, because "information" refers to a concept at its maximum compression. By definition, it can't be compressed further.
Take your point, I'm probably out of technical depth to throw these terms around on HN lightly.
What I was trying to say is that sometimes you need to provide additional context and information to those not using the same information model. Maybe it's not compression, but rather you have to include the 700kb of your information model with your message.
Information models aren't really something that one has, they're more of a semi-arbitrary frame of reference that we can choose. I'm assuming an information model where all public knowledge is accessible to everyone: you, me, your LLM, and my LLM. This means that conveying public knowledge doesn't convey any information (other than the fact that that knowledge is relevant to our conversation). This is a premise of my argument, and I'm fitting everything inside that frame of reference.
The LLMs don't have access to private information other than what you give it. If you give 300 bits of information to your LLM, the information it has is now "all public knowledge + 300 bits", and any text it generates is a subset of that information. By definition, it can't add any more information. If you give those 300 bits directly to me, I (and optionally my LLM if I want) now have "all public knowledge + 300 bits + my private thoughts", which is a superset of what your LLM has. Anything your LLM can infer can be inferred by me and my LLM. All your LLM can do is repackage that information into different text.
My opinion is that I don't get value out of that repackaging. I would rather read your packaging of those 300 bits rather than the LLM's packaging of those 300 bits.
For this current discussion, we could use an information model saying that your LLM has a knowledge base private to just you and it (e.g. local files or other conversations), and say that's separate from the prompt you give it. It sounds like maybe that's the information model you're thinking within.
In that frame of reference, maybe your shared knowledge base has 200 bits, and you type 100 bits into your LLM's prompt. My argument would still be that you're "giving" 300 bits to the LLM, and you should instead give them to me by sharing your knowledge base, or giving me the relevant information in your knowledge base in your own words. The latter option is definitely more work for you, and is the weakest point in my argument, but I'll still hold that preference.
With your narrow definition of the problem and reduction of communication down to pure utilitarian exchange of information between two parties that are willing to put in the effort to get the required context from the public domain - your argument is strong.
I don't believe that it holds true in context of typical communication between humans. You need to include the additional text to save people time looking things up, make the story flow better, fit in within the expected style of writing for intended audience etc.
To me the problem with LLM writing is not that someone is sending me 700kb of information I can get elsewhere, but that they didn't invest the time to understand what would be useful for me as a reader. I don't need 700kb of generic slop, but I probably need additional 300kb of relevant publicly available info to help me understand the purpose of the new information being sent.
In my opinion, not understanding the reader is exactly why bad writers remain bad writers - LLM doesn't know who the document is written for exactly.
hatthew · · focus · HN ↗
Writing is fundamentally the transfer of information from your brain to my brain. If you have 1000 bits of semantic information you want to transfer, you can't give 300 bits of semantic information to an LLM and have it fill in the remaining 700, because it doesn't know what those 700 bits are. If it's able to guess those 700 bits correctly, then they aren't true semantic information, and you really only have 300 bits you want to transfer. You might as well transfer those bits to me directly, rather than having the LLM add on an extra superfluous 700 bits that I then have to filter out.
entech · · focus · HN ↗
Can you explain to a child what it means when "GitHub is down" with less characters?
hatthew · · focus · HN ↗
entech · · focus · HN ↗
What I was trying to say is that sometimes you need to provide additional context and information to those not using the same information model. Maybe it's not compression, but rather you have to include the 700kb of your information model with your message.
hatthew · · focus · HN ↗
The LLMs don't have access to private information other than what you give it. If you give 300 bits of information to your LLM, the information it has is now "all public knowledge + 300 bits", and any text it generates is a subset of that information. By definition, it can't add any more information. If you give those 300 bits directly to me, I (and optionally my LLM if I want) now have "all public knowledge + 300 bits + my private thoughts", which is a superset of what your LLM has. Anything your LLM can infer can be inferred by me and my LLM. All your LLM can do is repackage that information into different text.
My opinion is that I don't get value out of that repackaging. I would rather read your packaging of those 300 bits rather than the LLM's packaging of those 300 bits.
For this current discussion, we could use an information model saying that your LLM has a knowledge base private to just you and it (e.g. local files or other conversations), and say that's separate from the prompt you give it. It sounds like maybe that's the information model you're thinking within.
In that frame of reference, maybe your shared knowledge base has 200 bits, and you type 100 bits into your LLM's prompt. My argument would still be that you're "giving" 300 bits to the LLM, and you should instead give them to me by sharing your knowledge base, or giving me the relevant information in your knowledge base in your own words. The latter option is definitely more work for you, and is the weakest point in my argument, but I'll still hold that preference.
entech · · focus · HN ↗
I don't believe that it holds true in context of typical communication between humans. You need to include the additional text to save people time looking things up, make the story flow better, fit in within the expected style of writing for intended audience etc.
To me the problem with LLM writing is not that someone is sending me 700kb of information I can get elsewhere, but that they didn't invest the time to understand what would be useful for me as a reader. I don't need 700kb of generic slop, but I probably need additional 300kb of relevant publicly available info to help me understand the purpose of the new information being sent.
In my opinion, not understanding the reader is exactly why bad writers remain bad writers - LLM doesn't know who the document is written for exactly.