On one side I agree that LLMs and Agents are not intelligent, but they present an illusion of intelligence given the shear amount of data they can process and act upon.
But that cannot be used to discredit the fact that these are incredibly powerful tools that can get out of control and cause great damage.
What is "intelligence" then. I think the universalism of LLMs is now evident enough that they can be considered "intelligent", even in slightly different realization compared to humans.
As of "great damage", I doubt it. They do not have self-preservation instinct (all "worrying" experiments are the attempts to initiate something resembling self-preservation from human initiative). The driving part is external - the query loop can always be turned off. So yes, a dangerous tool that can be exploited by humans (including governments, especially governments - which is why I am skeptical to government regulation proposals, particularly looking at what passes as governments in this era). But there are no inherent dangers from their own agency, as there is none.
>completely different realisation of intelligence to humans.
OK, so they are intelligent. I'm glad we agree.
The problem with this word is people associate intelligence with the process they realize is occurring inside their head which is only a tiny fraction of what intelligence encompasses. Humans (without study) know very little about intelligence and they very gray boundaries it has.
> As of "great damage", I doubt it. They do not have self-preservation instinct
The potential for great damage relies upon the lack of self-preservation instinct. That lack releases the so-called AI from an order under pain of death. Damage cannot be punished.
In my opinion, this article is talking about people who get into long conversations and become convinced the model is “intelligent”, maybe even “AGI”. This seems to be a trap people are falling into, even in 2026.
Initial definitions of AGI (as of 2014) are long surpassed. Then, AGI was defined as something that is general enough to work in many fields and orient themselves. It did not include being smarter than humans or even having comparable intelligence to humans. By original standards, we have AGI already for some time.
I have published a LLM-assisted proof of a long-standing problem in convex geometry (70 years) while not being a specialist in convex geometry. I am a scientist and I have education and scientific experience in computer science and systems biology, but I was never a mathematician. So, well, you can definitely work across fields at least.
Do you have a link? I would also be very interested in seeing the prompts, from what I've seen you still need to understand the field, maybe I'm wrong.
Okay? Maybe that's a specific use-case where they can possibly make progress. It's also likely that most of the leg-work for that specific proof has already been done and just a bit of recombination of leading theories and publications resulted in the answer.
If you have it ask for involved legal documents to give to a lawyer like 100% of the time they find problems with it. And someone who isn't in law would have not known any better.
When you have it write complex code that is not easy/quick to test, especially things that are specifically NOT concretely defined, like net-code (because it's all on the trade-offs you want to accept for your particular game), it's going to just repeatedly create sync issues.
I've asked it to write-up a detailed explanation of the different types of turns in 4-panel dance games and it's just permanently wrong no matter what I say.
The more unique and lacking of training data that exactly represents the problem statement the more impossible the statistical machine will generate text that makes sense.
The original article already attempted to do that but I'll take another stab. Humans tend to believe what they want to believe. They gain confidence when a confident person (or LLM) they work with exudes confidence. Where they might have hesitated or got stuck in analysis paralysis alone, the aid of a con (short for confidence) artist will get them moving. They then attribute the solution and praise for it solely to the confident person (or LLM) that nudged them along. Did the "psychic" who understands human motivation better than the cops actually help? Very likely. Did the psychic end up getting far more credit than they deserve (it's magical! You'll never believe! Well worth the money I spent!)? Absolutely.
The amount of real help vs. confidence/manipulation the LLM or con artist provides varies in each situation and scenario and mix of people (and LLMs) involved. It's something to strive to be aware and analytical about.
There is also something else horizontally to this that occurs.
There are numerous scientific discoveries that have been made because one human looked at all available data and turned off the assumptions other people in the field have been using for years, maybe hundreds of years. Once you delete the assumption and make a new one the answer is obvious any anyone from that point wonders how so many humans could have missed it for so much time.
This can occur readily with LLMs as it can with people. It's very likely we will see this a lot as the causal connection in the available data will connected differently in their minds.
Escaping from a local maximum can be very difficult as you have to climb uphill with a nearly infinite amount of freedom but no ability to see the horizon. Psychics, LLMs, or some guy name Bob walking in an providing a workable analogy for you to escape the local maxima and move closer to a global maxima are all the same. Moreso, if LLMs are AGI we should expect these "psychic" behaviors just as much as actual discovery because both continuums exist in the same problem space as human minds.
No they don't but psychics also don't have a verifiable feedback mechanism and unlimited retries.
Give the psychic the ability to adjust answers based on tests and you'll have have a fair fight.
atemerev · · focus · HN ↗
igortg · · focus · HN ↗
But that cannot be used to discredit the fact that these are incredibly powerful tools that can get out of control and cause great damage.
atemerev · · focus · HN ↗
As of "great damage", I doubt it. They do not have self-preservation instinct (all "worrying" experiments are the attempts to initiate something resembling self-preservation from human initiative). The driving part is external - the query loop can always be turned off. So yes, a dangerous tool that can be exploited by humans (including governments, especially governments - which is why I am skeptical to government regulation proposals, particularly looking at what passes as governments in this era). But there are no inherent dangers from their own agency, as there is none.
alansaber · · focus · HN ↗
pixl97 · · focus · HN ↗
OK, so they are intelligent. I'm glad we agree.
The problem with this word is people associate intelligence with the process they realize is occurring inside their head which is only a tiny fraction of what intelligence encompasses. Humans (without study) know very little about intelligence and they very gray boundaries it has.
chrisjj · · focus · HN ↗
The potential for great damage relies upon the lack of self-preservation instinct. That lack releases the so-called AI from an order under pain of death. Damage cannot be punished.
[deleted] · · focus · HN ↗
[deleted]
cmiles74 · · focus · HN ↗
atemerev · · focus · HN ↗
Current AGI definitions are intentionally vague.
krupan · · focus · HN ↗
- fawning over how amazing these tools are
- believing everything OpenAI and Anthropic say about how powerful and dangerous their product is
- minimizing the amount of human effort and involvement in every "AI" achievement
Madmallard · · focus · HN ↗
Everyone not in a particular field asking LLM about said field is rolling bad dice.
atemerev · · focus · HN ↗
tripledry · · focus · HN ↗
Madmallard · · focus · HN ↗
If you have it ask for involved legal documents to give to a lawyer like 100% of the time they find problems with it. And someone who isn't in law would have not known any better.
When you have it write complex code that is not easy/quick to test, especially things that are specifically NOT concretely defined, like net-code (because it's all on the trade-offs you want to accept for your particular game), it's going to just repeatedly create sync issues.
I've asked it to write-up a detailed explanation of the different types of turns in 4-panel dance games and it's just permanently wrong no matter what I say.
The more unique and lacking of training data that exactly represents the problem statement the more impossible the statistical machine will generate text that makes sense.
krupan · · focus · HN ↗
pixl97 · · focus · HN ↗
krupan · · focus · HN ↗
The amount of real help vs. confidence/manipulation the LLM or con artist provides varies in each situation and scenario and mix of people (and LLMs) involved. It's something to strive to be aware and analytical about.
pixl97 · · focus · HN ↗
There are numerous scientific discoveries that have been made because one human looked at all available data and turned off the assumptions other people in the field have been using for years, maybe hundreds of years. Once you delete the assumption and make a new one the answer is obvious any anyone from that point wonders how so many humans could have missed it for so much time.
This can occur readily with LLMs as it can with people. It's very likely we will see this a lot as the causal connection in the available data will connected differently in their minds.
Escaping from a local maximum can be very difficult as you have to climb uphill with a nearly infinite amount of freedom but no ability to see the horizon. Psychics, LLMs, or some guy name Bob walking in an providing a workable analogy for you to escape the local maxima and move closer to a global maxima are all the same. Moreso, if LLMs are AGI we should expect these "psychic" behaviors just as much as actual discovery because both continuums exist in the same problem space as human minds.
_superposition_ · · focus · HN ↗