At what point is human intelligence going to hold back machine intelligence?
Imagine you are evaluating what the machine should do when it is improving itself. It does a bunch of work and returns with "I supervaluated the liminal overdecomposition from the previous homological calibulation pass. It shows us that subtransitory mulutination will underspecify the tensor of stermullification. Where do you want to go from here?"
It will be like when you are reading a Wikipedia about a topic you don't understand. You follow the links, and you get more questions with more links. Your whole day is taken up following links, to the point where you forgot the original question.
Except this time, all the words come from the AI's work. You can't refer to an external authority who has already been there and can tell you what to do.
The AI needs you to tell it whether it is more intelligent than it was before, but you don't know, because you can't follow its reasoning any more. It's like an ordinary person trying to hire a math professor, there's just no way to do it.
But whereas a human math prof can evaluate another one, a machine intelligence can't evaluate another one, by construction. Because it's still usefulness to humans that is the evaluation criterion.
At that point, the machines correctness doesn't need to be evaluated by humans, it just needs to provide a recipe for how to achieve some process.
Is there a way for an llm to coin a word, and absorb it into its model? During training maybe… but not after - not the way they’re designed now, anyway.
For it to have new vocabulary we dont understand, it needs to have novel ideas that need words coined for them, and a way to persist those ideas and words into the future. I don’t think that exists.
To me this hypothetical make it clear this won’t happen, not unless there are fundamental changes to what llms are. It doesn’t suggest it will happen. To me, anyway.
The decoding step (output of final layer -> word) is not strictly needed. You can feed the output directly into the next layer (Chain of Continuous Thought). You can 'decode' the output into things other than words.
How does that give the llm a way to speak in meaning that humans can’t understand?
You input text into the system, that moves through layers as chains of weights. Sure it doesn’t ever have to be output as text, but for the text-in-text-out machine to have any impact on the world around it, text must be generated.
I don't see why this couldn't be possible. We use LLMs whose weights are frozen and are not updated at inference, most likely this is due to reasons of cost, stability and control.
Theoretically you could update the weights at inference time too though so the model evolved as it's used. Surely some people are trying this already.
This is a very outdated view on what an LLM is and how it works. We are way past the "stochastic parrot" phase, ever since double descent and proper generalisation. Then with the various flavours of RL the models learn to pluck patterns / circuits out of the massive data and combine them on the fly. There's absolutely no reason to think they can't "invent" new words, because words are just combinations of tokens at the end of the day. So if they can come up with "in this codebase bar is load-bearing" they can similarly come up with "bumblespin is the new word for reversing the polarity of the quantum surface of a spin-aware brane in four dimensional bumblespace".
So you’ve claimed that they have half of my precondition for them to speak in ways that we don’t understand.
Coining a term on the fly: yes
Remembering that term and incorporating it into its mind: no.
But, beyond saying they aren’t stochastic parrots you haven’t really given anything to substantiate the idea that they aren’t (very complex) stochastic parrots.
Interesting. I don’t have a counter to the coining a new term evidence in that article. Potentially that’s solid evidence of the first aspect.
But I do have a counter for the second point:
Context is not a part of the model. It’s an ephemeral blend of input and output fed back in as input.
Online learning is potentially a way for incorporating and evolving coined terms to happen. But since that’s not what any of the models are doing, something would have to change in how we do things before it could happen.
The model will have to convince the human that it's making the right kind of progress. That will necessarily become part of the improvement loop - either implicitly (human trusts RSI) or explicitly (human gatekeeps every major decision).
I love your thought experiment. May I counter, what purpose does such a machine have to us, that can think beyond our needs? Sorry, but to reference the great Rick and Morty, "your purpose is to pass the butter".
“To us” is pivotal there. Continuing GP’s thought experiment: what if the model that produced the output perceives that the human it was presented to offers no value in helping it learn further?
I think if an AI developed completely new fields of thought or science.
But given our current relationship even if it did I can't foresee a point where it couldn't walk us through the necessary steps or supply the pros/cons for whatever problem is being addressed.
2 issues - trying to understand how it came to its conclusion because I feel true AI has got to be non-human intelligence. Or a something catastrophic happens and we as a species are back in the stone age. Imagine today's AI trying to converse with a cave man (yes, one without modern languages even).
> Imagine you are evaluating what the machine should do when it is improving itself. It does a bunch of work and returns with "... Where do you want to go from here?"
It's worth pointing out that so many people already see and use AI the same way. In such a situation some would ask AI to provide options, and they would choose and experiment with those options. Of course, contexts/stakes can be vastly different.
Given that there are so many phenomena in nature that we can't explain or fully understand which doesn't prevent them from existing or being useful, there could be a future where humans accept the same about the things AI comes up with as long as this leads to desired outcomes. We still might have names for them, but our brain thinking/knowledge capacity wouldn't allow us to fully comprehend them. We'd need frameworks/systems in place to turn these AI features/decisions on/off, although it's hard to imagine how this wouldn't increase the likelihood of something going out of control.
lordnacho · · focus · HN ↗
Imagine you are evaluating what the machine should do when it is improving itself. It does a bunch of work and returns with "I supervaluated the liminal overdecomposition from the previous homological calibulation pass. It shows us that subtransitory mulutination will underspecify the tensor of stermullification. Where do you want to go from here?"
It will be like when you are reading a Wikipedia about a topic you don't understand. You follow the links, and you get more questions with more links. Your whole day is taken up following links, to the point where you forgot the original question.
Except this time, all the words come from the AI's work. You can't refer to an external authority who has already been there and can tell you what to do.
The AI needs you to tell it whether it is more intelligent than it was before, but you don't know, because you can't follow its reasoning any more. It's like an ordinary person trying to hire a math professor, there's just no way to do it.
But whereas a human math prof can evaluate another one, a machine intelligence can't evaluate another one, by construction. Because it's still usefulness to humans that is the evaluation criterion.
ludston · · focus · HN ↗
kennywinker · · focus · HN ↗
For it to have new vocabulary we dont understand, it needs to have novel ideas that need words coined for them, and a way to persist those ideas and words into the future. I don’t think that exists.
To me this hypothetical make it clear this won’t happen, not unless there are fundamental changes to what llms are. It doesn’t suggest it will happen. To me, anyway.
skew-aberration · · focus · HN ↗
kennywinker · · focus · HN ↗
You input text into the system, that moves through layers as chains of weights. Sure it doesn’t ever have to be output as text, but for the text-in-text-out machine to have any impact on the world around it, text must be generated.
skew-aberration · · focus · HN ↗
We could find common vectors in the space of common output tokens, which would serve as new words.
We could decode the output tokens into other actions (like a boolean output 'is safe / is unsafe input').
hereonout2 · · focus · HN ↗
Theoretically you could update the weights at inference time too though so the model evolved as it's used. Surely some people are trying this already.
NitpickLawyer · · focus · HN ↗
kennywinker · · focus · HN ↗
Coining a term on the fly: yes
Remembering that term and incorporating it into its mind: no.
But, beyond saying they aren’t stochastic parrots you haven’t really given anything to substantiate the idea that they aren’t (very complex) stochastic parrots.
hereonout2 · · focus · HN ↗
<a href="https://www.euronews.com/2026/09/16/ai-chatbots-developed-a-secret-language-that-baffled-humans-study-says" rel="nofollow">https://www.euronews.com/2026/09/16/ai-chatbots-developed-a-...
Remembering that term and incorporating it into its mind:
You're ignoring simple things like a growing context but also the possibility of online learning
kennywinker · · focus · HN ↗
But I do have a counter for the second point:
Context is not a part of the model. It’s an ephemeral blend of input and output fed back in as input.
Online learning is potentially a way for incorporating and evolving coined terms to happen. But since that’s not what any of the models are doing, something would have to change in how we do things before it could happen.
skew-aberration · · focus · HN ↗
telesilla · · focus · HN ↗
onion2k · · focus · HN ↗
We can think of questions we can't answer. It can answer them.
RandomLensman · · focus · HN ↗
spinningslate · · focus · HN ↗
vasusai · · focus · HN ↗
But given our current relationship even if it did I can't foresee a point where it couldn't walk us through the necessary steps or supply the pros/cons for whatever problem is being addressed.
2 issues - trying to understand how it came to its conclusion because I feel true AI has got to be non-human intelligence. Or a something catastrophic happens and we as a species are back in the stone age. Imagine today's AI trying to converse with a cave man (yes, one without modern languages even).
rpozarickij · · focus · HN ↗
It's worth pointing out that so many people already see and use AI the same way. In such a situation some would ask AI to provide options, and they would choose and experiment with those options. Of course, contexts/stakes can be vastly different.
Given that there are so many phenomena in nature that we can't explain or fully understand which doesn't prevent them from existing or being useful, there could be a future where humans accept the same about the things AI comes up with as long as this leads to desired outcomes. We still might have names for them, but our brain thinking/knowledge capacity wouldn't allow us to fully comprehend them. We'd need frameworks/systems in place to turn these AI features/decisions on/off, although it's hard to imagine how this wouldn't increase the likelihood of something going out of control.