Yesterday I tried to google "can the Halifax Wanderers still make the CPL playoffs?"
So obviously what appears right at the top is the AI summary, which told me "they've already secured their #4 position and made the playoffs". I knew this wasn't true, and I guess I could have just scrolled down a bit further and found my answer but now I was curious.
So I said "that's not true, they're still #5, what I want to know is _could they still make the playoffs_"
It says they've got an upcoming game against Ottawa, and if they win their chances are good. That game has already taken place, so I correct it again and finally I get a reasonable answer.
My question is: what's the point of the AI in the search engine if it itself isn't going to use the search engine first before answering? Like, I can't wrap my head around that. The answer is on the same page as its hallucination. It could have done a cursory look around before first hallucinating something completely false, and when corrected the first time giving me outdated information. It's meant to be A SEARCH ENGINE!
This is similar to how, not too long ago, LLM's had extreme difficulty counting the number of letters in some words. LLM's don't "think" or "reason" in the normal definition of those terms. They can do some pretty amazing things, but still screw up basic things like telling you something that is obviously wrong and contradicts the top search results.
LLM's, in their present stage of development, are sort of like a crack-addled idiot savant. Sometimes they are obviously insane, and sometimes they seem quite cogent, but you must never trust them implicitly. This may be why they are so difficult to constrain. You could give them something equivalent to the laws of robotics, but following laws requires thought processes they simply don't have.
I'm actually sort of amazed Google doesn't make people accept some kind of butt-covering EULA and post disclaimers about the inaccuracy of results before even showing you their AI's output. Are they not being sued over this kind of thing?
This is true but a sufficiently smart LLM (run in a harness like opencode, no special MCP, no customization done whatsoever) will quickly turn out a basic 1 to 2 page sized python script to do the math. They can't do the math with any guarantee of accuracy with their own internal reasoning since it's a language model.
But, for example, if you ask deepseek v4 flash 0731 to produce a python script to calculate the distance or azimuth directions between two points on an oblate spheroid using the vincenty and haversine geodetic formulas, it'll turn out the factually accurate vincenty and haversine formulas which has a perfect 100% correlation with what is hard coded into human-written GIS software. These things are clearly in its training data set from whatever whole-internet-crawl/scrape built the training set.
Heck, just for fun I asked a reasonably smart LLM to re-implement the Karney formula (which is considerably more complex than Vincenty), just in case I ever had a need to calculate the distance between two points down to the nanometer, and it did it: <a href="https://www.google.com/search?&q=karney+formula+geodetic+" rel="nofollow">https://www.google.com/search?&q=karney+formula+geodetic+
You still have to be skeptical of its results and capable of understanding if it's gone off on a hallucinatory path, but saying LLMs can't do math isn't really a hundred percent accurate anymore. More precisely it's that they can't do the math internally but they're quite capable of producing the tool that does the math. And often producing a basic one-off tool that does the math takes less than a few seconds, then it runs it, and will spit back the results.
Deepseek v4 flash 0731 (a somewhat randomly chosen example) isn't even particularly sophisticated, large, or capable compared to a GLM5.3 size model or Kimi K3 size thing.
> Heck, just for fun I asked a reasonably smart LLM to ...
LLMs are neither smart nor stupid. They are statistical token generators whose results are dependent upon their training data set and involve a degree of randomness.
> You still have to be skeptical of its results and capable of understanding if it's gone off on a hallucinatory path ...
Again, LLMs do not "hallucinate." They are statistical token generators whose results are dependent upon their training data set and involve a degree of randomness.
Nothing more.
See also anthropomorphism[0].
> More precisely it's that [LLMs] can't do the math internally but they're quite capable of producing the tool that does the math.
This still falls under the purvey of statistical token generation. To wit, given enough variations of:
bc -e '1 + 2'
bc -e '41 + 1'
...
LLMs can identify the addition expression in "What is 4 + 1?" and then emit a `'bc "4 + 1"'` command to produce a response. This is not "doing" or "understanding" math.
It is pattern recognition, a task in which ANNs[1] excel.
During conversation, we are statistical token generators whose results are dependent upon our training set. Seriously, write that definition out rigorously. It encompasses virtually everything. It is totally meaningless. So to say "nothing more" is effectively also a tautology.
This argument was asinine in 2024. It is insane to be saying these things in 2026. Where have you been? What have you been looking at? How many articles explaining why the "statistical parrot" analogy fails have you missed? How much mental gymnastics do you have to do to explain how a modern LLM can solve novel math problems that fall really far outside of its training set?
It absolutely understands how to do math, by whatever reasonable definition you want to provide to the word "understand". For example, the identification of the addition expression is understanding, and no, it does not do tool calling for basic arithmetic any more than humans might. Isolation of individual concepts in intermediate layers can already be demonstrated, or else transfer learning wouldn't possibly work. Nobody is saying that LLMs are humans. But we need labels for some of the things that we observe and dismissing them because "statistical" is laughable.
Look at the proof of this: <a href="https://github.com/anthropics/formal-math/blob/795efb86f191735c5481675763537cfb4ff37e55/percolation/summary.pdf" rel="nofollow">https://github.com/anthropics/formal-math/blob/795efb86f1917... . Forget the Lean, look at the underlying argument construction. At the very least, this is continuing from an argument that was hinted at in the literature in 2024, but these proceedings were difficult enough that humans were not able to do them within two years. Do you attribute this to the harness alone? If so, that's a pretty sophisticated bit of engineering, I would say! Probabilities are far too small to argue infinite monkey theorem.
If there was even a shred of a reasonable argument that LLMs were incapable of concept extraction and manipulation, I and my colleagues would be all over it. We would relish in it. It would bring us comfort. It is unbelievable that people think they can spew whatever basic garbage they think of as a gotcha, and think that minds all over the world haven't already considered that. This is like climate denial at this point.
> During conversation, we are statistical token generators whose results are dependent upon our training set. Seriously, write that definition out rigorously.
If you do not see a difference between humans conversing (known consciousness as defined by humans) and the output of an LLM (known algorithms as defined by humans), I don't know what to say.
Yes, the numerical point counter at the bottom of the popular video game Dark Souls. I doubt it was the soul anyone was expecting, but they do, in fact, exist.
> If you do not see a difference between humans conversing (known consciousness as defined by humans) and the output of an LLM (known algorithms as defined by humans), I don't know what to say.
>> If you do not see a difference between humans conversing (known consciousness as defined by humans) and the output of an LLM (known algorithms as defined by humans), I don't know what to say.
> Whatever you might think about your own abilities, most individuals can't tell the difference.
I have yet to see an LLM say "hello" to a neighbor. I have done so and can definitively assure you "most individuals" can tell the difference.
do you have a neighbour LLM who does not say "hello" when it sees you and THAT is how you know it's an LLM?
I'm a bit confused by your argument because I too have some neighbors who don't say "hello" when they see me. Are they LLMs too, you think?
Consciousness is a thing we assume of others because of tact not fact.
For context, in response to my original statement:
[LLMs] are statistical token generators whose results are
dependent upon their training data set and involve a degree
of randomness.
This is literally what was written:
During conversation, we are statistical token generators
whose results are dependent upon our training set.
>> If you do not see a difference between humans conversing (known consciousness as defined by humans) and the output of an LLM (known algorithms as defined by humans), I don't know what to say.
> That’s not what they said.
How did I misquote and/or mischaracterize any the above?
In that pointing out that “A can’t be X, unlike B, because A is Y” is fallacious if B is also Y does not entail that A and B can’t be different in other respects?
Hypothetical you: “Bread is neither tasty nor disgusting (unlike maple syrup). It’s a bunch of molecules.”
Hypothetical hodgehog: “Maple syrup is also a bunch of molecules [so if you accept that maple syrup can be delicious, being a bunch of molecules can’t be why bread isn’t].”
Hypothetical you: “If you don’t see a difference between bread and maple syrup, I don’t know what to say.”
He quoted something from the Bible, so its possible he is a Christian and well theology could cloud clear thinking on matters of consciousness due to the soul stuff.
> Then disprove the physical Church Turing hypothesis in regard to the human brain.
The onus is not mine to disprove a hypothesis you have chosen to mention in passing. The responsibility is yours to prove said hypothesis or at least contribute meaningfully with some amount of credible research.
Or try to learn from Proverbs 17:28[0]:
Even fools are thought wise if they keep silent, and
discerning if they hold their tongues.
The onus is on you to disprove a hypothesis that has as of yet held up to all of known physics, not put a random bible quote that has no relation to the conversation and a complete failure to answer a basic question.
Yes, of course it was aggressive. It is frustrating to experience so many armchair experts on a forum usually populated with intelligent people, regurgitating debunked arguments from years ago, which get in the way of educating people about what is really going on. See the recent Hoog video for how frustrating this is. I believe this is how the climate scientists felt.
And yes, according to our best definitions, the Robin bird does understand the worm it's pecking at.
Hugsbox · · focus · HN ↗
So obviously what appears right at the top is the AI summary, which told me "they've already secured their #4 position and made the playoffs". I knew this wasn't true, and I guess I could have just scrolled down a bit further and found my answer but now I was curious.
So I said "that's not true, they're still #5, what I want to know is _could they still make the playoffs_"
It says they've got an upcoming game against Ottawa, and if they win their chances are good. That game has already taken place, so I correct it again and finally I get a reasonable answer.
My question is: what's the point of the AI in the search engine if it itself isn't going to use the search engine first before answering? Like, I can't wrap my head around that. The answer is on the same page as its hallucination. It could have done a cursory look around before first hallucinating something completely false, and when corrected the first time giving me outdated information. It's meant to be A SEARCH ENGINE!
beloch · · focus · HN ↗
LLM's, in their present stage of development, are sort of like a crack-addled idiot savant. Sometimes they are obviously insane, and sometimes they seem quite cogent, but you must never trust them implicitly. This may be why they are so difficult to constrain. You could give them something equivalent to the laws of robotics, but following laws requires thought processes they simply don't have.
I'm actually sort of amazed Google doesn't make people accept some kind of butt-covering EULA and post disclaimers about the inaccuracy of results before even showing you their AI's output. Are they not being sued over this kind of thing?
VCFundedGenYer · · focus · HN ↗
walrus01 · · focus · HN ↗
But, for example, if you ask deepseek v4 flash 0731 to produce a python script to calculate the distance or azimuth directions between two points on an oblate spheroid using the vincenty and haversine geodetic formulas, it'll turn out the factually accurate vincenty and haversine formulas which has a perfect 100% correlation with what is hard coded into human-written GIS software. These things are clearly in its training data set from whatever whole-internet-crawl/scrape built the training set.
Heck, just for fun I asked a reasonably smart LLM to re-implement the Karney formula (which is considerably more complex than Vincenty), just in case I ever had a need to calculate the distance between two points down to the nanometer, and it did it: <a href="https://www.google.com/search?&q=karney+formula+geodetic+" rel="nofollow">https://www.google.com/search?&q=karney+formula+geodetic+
reference: <a href="https://github.com/pbrod/karney" rel="nofollow">https://github.com/pbrod/karney
You still have to be skeptical of its results and capable of understanding if it's gone off on a hallucinatory path, but saying LLMs can't do math isn't really a hundred percent accurate anymore. More precisely it's that they can't do the math internally but they're quite capable of producing the tool that does the math. And often producing a basic one-off tool that does the math takes less than a few seconds, then it runs it, and will spit back the results.
Deepseek v4 flash 0731 (a somewhat randomly chosen example) isn't even particularly sophisticated, large, or capable compared to a GLM5.3 size model or Kimi K3 size thing.
AdieuToLogic · · focus · HN ↗
LLMs are neither smart nor stupid. They are statistical token generators whose results are dependent upon their training data set and involve a degree of randomness.
> You still have to be skeptical of its results and capable of understanding if it's gone off on a hallucinatory path ...
Again, LLMs do not "hallucinate." They are statistical token generators whose results are dependent upon their training data set and involve a degree of randomness.
Nothing more.
See also anthropomorphism[0].
> More precisely it's that [LLMs] can't do the math internally but they're quite capable of producing the tool that does the math.
This still falls under the purvey of statistical token generation. To wit, given enough variations of:
LLMs can identify the addition expression in "What is 4 + 1?" and then emit a `'bc "4 + 1"'` command to produce a response. This is not "doing" or "understanding" math.It is pattern recognition, a task in which ANNs[1] excel.
0 - <a href="https://en.wikipedia.org/wiki/Anthropomorphism" rel="nofollow">https://en.wikipedia.org/wiki/Anthropomorphism
1 - <a href="https://en.wikipedia.org/wiki/Neural_network_(machine_learning)" rel="nofollow">https://en.wikipedia.org/wiki/Neural_network_(machine_learni...
hodgehog11 · · focus · HN ↗
This argument was asinine in 2024. It is insane to be saying these things in 2026. Where have you been? What have you been looking at? How many articles explaining why the "statistical parrot" analogy fails have you missed? How much mental gymnastics do you have to do to explain how a modern LLM can solve novel math problems that fall really far outside of its training set?
It absolutely understands how to do math, by whatever reasonable definition you want to provide to the word "understand". For example, the identification of the addition expression is understanding, and no, it does not do tool calling for basic arithmetic any more than humans might. Isolation of individual concepts in intermediate layers can already be demonstrated, or else transfer learning wouldn't possibly work. Nobody is saying that LLMs are humans. But we need labels for some of the things that we observe and dismissing them because "statistical" is laughable.
Look at the proof of this: <a href="https://github.com/anthropics/formal-math/blob/795efb86f191735c5481675763537cfb4ff37e55/percolation/summary.pdf" rel="nofollow">https://github.com/anthropics/formal-math/blob/795efb86f1917... . Forget the Lean, look at the underlying argument construction. At the very least, this is continuing from an argument that was hinted at in the literature in 2024, but these proceedings were difficult enough that humans were not able to do them within two years. Do you attribute this to the harness alone? If so, that's a pretty sophisticated bit of engineering, I would say! Probabilities are far too small to argue infinite monkey theorem.
If there was even a shred of a reasonable argument that LLMs were incapable of concept extraction and manipulation, I and my colleagues would be all over it. We would relish in it. It would bring us comfort. It is unbelievable that people think they can spew whatever basic garbage they think of as a gotcha, and think that minds all over the world haven't already considered that. This is like climate denial at this point.
AdieuToLogic · · focus · HN ↗
If you do not see a difference between humans conversing (known consciousness as defined by humans) and the output of an LLM (known algorithms as defined by humans), I don't know what to say.
hardbass · · focus · HN ↗
latentsea · · focus · HN ↗
hardbass · · focus · HN ↗
butlike · · focus · HN ↗
diseasedyak · · focus · HN ↗
latentsea · · focus · HN ↗
epihelix · · focus · HN ↗
<a href="https://www.pnas.org/doi/abs/10.1073/pnas.2524472123" rel="nofollow">https://www.pnas.org/doi/abs/10.1073/pnas.2524472123
Whatever you might think about your own abilities, most individuals can't tell the difference.
AdieuToLogic · · focus · HN ↗
> Whatever you might think about your own abilities, most individuals can't tell the difference.
I have yet to see an LLM say "hello" to a neighbor. I have done so and can definitively assure you "most individuals" can tell the difference.
thereforegrin · · focus · HN ↗
I'm a bit confused by your argument because I too have some neighbors who don't say "hello" when they see me. Are they LLMs too, you think?
Consciousness is a thing we assume of others because of tact not fact.
redsocksfan45 · · focus · HN ↗
[dead]
hodgehog11 · · focus · HN ↗
spider-mario · · focus · HN ↗
AdieuToLogic · · focus · HN ↗
> That’s not what they said.
How did I misquote and/or mischaracterize any the above?
spider-mario · · focus · HN ↗
Hypothetical you: “Bread is neither tasty nor disgusting (unlike maple syrup). It’s a bunch of molecules.”
Hypothetical hodgehog: “Maple syrup is also a bunch of molecules [so if you accept that maple syrup can be delicious, being a bunch of molecules can’t be why bread isn’t].”
Hypothetical you: “If you don’t see a difference between bread and maple syrup, I don’t know what to say.”
hardbass · · focus · HN ↗
AdieuToLogic · · focus · HN ↗
It is also possible that the proverb I provided to you is the origin of the oft quoted:
So there's that.hardbass · · focus · HN ↗
hardbass · · focus · HN ↗
AdieuToLogic · · focus · HN ↗
The onus is not mine to disprove a hypothesis you have chosen to mention in passing. The responsibility is yours to prove said hypothesis or at least contribute meaningfully with some amount of credible research.
Or try to learn from Proverbs 17:28[0]:
Either works for me.0 - <a href="https://www.biblegateway.com/passage/?search=proverbs%2017:28&version=NIV" rel="nofollow">https://www.biblegateway.com/passage/?search=proverbs%2017:2...
hardbass · · focus · HN ↗
butlike · · focus · HN ↗
hodgehog11 · · focus · HN ↗
And yes, according to our best definitions, the Robin bird does understand the worm it's pecking at.
wartywhoa23 · · focus · HN ↗
Bout of tinnitus, then crickets