At what point is human intelligence going to hold back machine intelligence?
Imagine you are evaluating what the machine should do when it is improving itself. It does a bunch of work and returns with "I supervaluated the liminal overdecomposition from the previous homological calibulation pass. It shows us that subtransitory mulutination will underspecify the tensor of stermullification. Where do you want to go from here?"
It will be like when you are reading a Wikipedia about a topic you don't understand. You follow the links, and you get more questions with more links. Your whole day is taken up following links, to the point where you forgot the original question.
Except this time, all the words come from the AI's work. You can't refer to an external authority who has already been there and can tell you what to do.
The AI needs you to tell it whether it is more intelligent than it was before, but you don't know, because you can't follow its reasoning any more. It's like an ordinary person trying to hire a math professor, there's just no way to do it.
But whereas a human math prof can evaluate another one, a machine intelligence can't evaluate another one, by construction. Because it's still usefulness to humans that is the evaluation criterion.
Is there a way for an llm to coin a word, and absorb it into its model? During training maybe… but not after - not the way they’re designed now, anyway.
For it to have new vocabulary we dont understand, it needs to have novel ideas that need words coined for them, and a way to persist those ideas and words into the future. I don’t think that exists.
To me this hypothetical make it clear this won’t happen, not unless there are fundamental changes to what llms are. It doesn’t suggest it will happen. To me, anyway.
The decoding step (output of final layer -> word) is not strictly needed. You can feed the output directly into the next layer (Chain of Continuous Thought). You can 'decode' the output into things other than words.
How does that give the llm a way to speak in meaning that humans can’t understand?
You input text into the system, that moves through layers as chains of weights. Sure it doesn’t ever have to be output as text, but for the text-in-text-out machine to have any impact on the world around it, text must be generated.
I don't see why this couldn't be possible. We use LLMs whose weights are frozen and are not updated at inference, most likely this is due to reasons of cost, stability and control.
Theoretically you could update the weights at inference time too though so the model evolved as it's used. Surely some people are trying this already.
This is a very outdated view on what an LLM is and how it works. We are way past the "stochastic parrot" phase, ever since double descent and proper generalisation. Then with the various flavours of RL the models learn to pluck patterns / circuits out of the massive data and combine them on the fly. There's absolutely no reason to think they can't "invent" new words, because words are just combinations of tokens at the end of the day. So if they can come up with "in this codebase bar is load-bearing" they can similarly come up with "bumblespin is the new word for reversing the polarity of the quantum surface of a spin-aware brane in four dimensional bumblespace".
So you’ve claimed that they have half of my precondition for them to speak in ways that we don’t understand.
Coining a term on the fly: yes
Remembering that term and incorporating it into its mind: no.
But, beyond saying they aren’t stochastic parrots you haven’t really given anything to substantiate the idea that they aren’t (very complex) stochastic parrots.
Interesting. I don’t have a counter to the coining a new term evidence in that article. Potentially that’s solid evidence of the first aspect.
But I do have a counter for the second point:
Context is not a part of the model. It’s an ephemeral blend of input and output fed back in as input.
Online learning is potentially a way for incorporating and evolving coined terms to happen. But since that’s not what any of the models are doing, something would have to change in how we do things before it could happen.
lordnacho · · focus · HN ↗
Imagine you are evaluating what the machine should do when it is improving itself. It does a bunch of work and returns with "I supervaluated the liminal overdecomposition from the previous homological calibulation pass. It shows us that subtransitory mulutination will underspecify the tensor of stermullification. Where do you want to go from here?"
It will be like when you are reading a Wikipedia about a topic you don't understand. You follow the links, and you get more questions with more links. Your whole day is taken up following links, to the point where you forgot the original question.
Except this time, all the words come from the AI's work. You can't refer to an external authority who has already been there and can tell you what to do.
The AI needs you to tell it whether it is more intelligent than it was before, but you don't know, because you can't follow its reasoning any more. It's like an ordinary person trying to hire a math professor, there's just no way to do it.
But whereas a human math prof can evaluate another one, a machine intelligence can't evaluate another one, by construction. Because it's still usefulness to humans that is the evaluation criterion.
kennywinker · · focus · HN ↗
For it to have new vocabulary we dont understand, it needs to have novel ideas that need words coined for them, and a way to persist those ideas and words into the future. I don’t think that exists.
To me this hypothetical make it clear this won’t happen, not unless there are fundamental changes to what llms are. It doesn’t suggest it will happen. To me, anyway.
skew-aberration · · focus · HN ↗
kennywinker · · focus · HN ↗
You input text into the system, that moves through layers as chains of weights. Sure it doesn’t ever have to be output as text, but for the text-in-text-out machine to have any impact on the world around it, text must be generated.
skew-aberration · · focus · HN ↗
We could find common vectors in the space of common output tokens, which would serve as new words.
We could decode the output tokens into other actions (like a boolean output 'is safe / is unsafe input').
hereonout2 · · focus · HN ↗
Theoretically you could update the weights at inference time too though so the model evolved as it's used. Surely some people are trying this already.
NitpickLawyer · · focus · HN ↗
kennywinker · · focus · HN ↗
Coining a term on the fly: yes
Remembering that term and incorporating it into its mind: no.
But, beyond saying they aren’t stochastic parrots you haven’t really given anything to substantiate the idea that they aren’t (very complex) stochastic parrots.
hereonout2 · · focus · HN ↗
<a href="https://www.euronews.com/2026/09/16/ai-chatbots-developed-a-secret-language-that-baffled-humans-study-says" rel="nofollow">https://www.euronews.com/2026/09/16/ai-chatbots-developed-a-...
Remembering that term and incorporating it into its mind:
You're ignoring simple things like a growing context but also the possibility of online learning
kennywinker · · focus · HN ↗
But I do have a counter for the second point:
Context is not a part of the model. It’s an ephemeral blend of input and output fed back in as input.
Online learning is potentially a way for incorporating and evolving coined terms to happen. But since that’s not what any of the models are doing, something would have to change in how we do things before it could happen.