Yesterday I tried to google "can the Halifax Wanderers still make the CPL playoffs?"
So obviously what appears right at the top is the AI summary, which told me "they've already secured their #4 position and made the playoffs". I knew this wasn't true, and I guess I could have just scrolled down a bit further and found my answer but now I was curious.
So I said "that's not true, they're still #5, what I want to know is _could they still make the playoffs_"
It says they've got an upcoming game against Ottawa, and if they win their chances are good. That game has already taken place, so I correct it again and finally I get a reasonable answer.
My question is: what's the point of the AI in the search engine if it itself isn't going to use the search engine first before answering? Like, I can't wrap my head around that. The answer is on the same page as its hallucination. It could have done a cursory look around before first hallucinating something completely false, and when corrected the first time giving me outdated information. It's meant to be A SEARCH ENGINE!
This is similar to how, not too long ago, LLM's had extreme difficulty counting the number of letters in some words. LLM's don't "think" or "reason" in the normal definition of those terms. They can do some pretty amazing things, but still screw up basic things like telling you something that is obviously wrong and contradicts the top search results.
LLM's, in their present stage of development, are sort of like a crack-addled idiot savant. Sometimes they are obviously insane, and sometimes they seem quite cogent, but you must never trust them implicitly. This may be why they are so difficult to constrain. You could give them something equivalent to the laws of robotics, but following laws requires thought processes they simply don't have.
I'm actually sort of amazed Google doesn't make people accept some kind of butt-covering EULA and post disclaimers about the inaccuracy of results before even showing you their AI's output. Are they not being sued over this kind of thing?
This is true but a sufficiently smart LLM (run in a harness like opencode, no special MCP, no customization done whatsoever) will quickly turn out a basic 1 to 2 page sized python script to do the math. They can't do the math with any guarantee of accuracy with their own internal reasoning since it's a language model.
But, for example, if you ask deepseek v4 flash 0731 to produce a python script to calculate the distance or azimuth directions between two points on an oblate spheroid using the vincenty and haversine geodetic formulas, it'll turn out the factually accurate vincenty and haversine formulas which has a perfect 100% correlation with what is hard coded into human-written GIS software. These things are clearly in its training data set from whatever whole-internet-crawl/scrape built the training set.
Heck, just for fun I asked a reasonably smart LLM to re-implement the Karney formula (which is considerably more complex than Vincenty), just in case I ever had a need to calculate the distance between two points down to the nanometer, and it did it: <a href="https://www.google.com/search?&q=karney+formula+geodetic+" rel="nofollow">https://www.google.com/search?&q=karney+formula+geodetic+
You still have to be skeptical of its results and capable of understanding if it's gone off on a hallucinatory path, but saying LLMs can't do math isn't really a hundred percent accurate anymore. More precisely it's that they can't do the math internally but they're quite capable of producing the tool that does the math. And often producing a basic one-off tool that does the math takes less than a few seconds, then it runs it, and will spit back the results.
Deepseek v4 flash 0731 (a somewhat randomly chosen example) isn't even particularly sophisticated, large, or capable compared to a GLM5.3 size model or Kimi K3 size thing.
> Heck, just for fun I asked a reasonably smart LLM to ...
LLMs are neither smart nor stupid. They are statistical token generators whose results are dependent upon their training data set and involve a degree of randomness.
> You still have to be skeptical of its results and capable of understanding if it's gone off on a hallucinatory path ...
Again, LLMs do not "hallucinate." They are statistical token generators whose results are dependent upon their training data set and involve a degree of randomness.
Nothing more.
See also anthropomorphism[0].
> More precisely it's that [LLMs] can't do the math internally but they're quite capable of producing the tool that does the math.
This still falls under the purvey of statistical token generation. To wit, given enough variations of:
bc -e '1 + 2'
bc -e '41 + 1'
...
LLMs can identify the addition expression in "What is 4 + 1?" and then emit a `'bc "4 + 1"'` command to produce a response. This is not "doing" or "understanding" math.
It is pattern recognition, a task in which ANNs[1] excel.
> They are statistical token generators whose results are dependent upon their training data set and involve a degree of randomness.
You haven't demonstrated why this matters.
> Nothing more.
Are you contending that complex systems cannot be more than the sum of their parts?
A market is nothing more than offers and counter offers.
A ant colony is nothing more than scent trails.
All life on earth is nothing more than reproduction with variation.
> This still falls under the purvey of statistical token generation.
Stating the mechanism does nothing to provide insight into capability. For instance: a nuclear power plant boils water by using fuel rods for heat. What does that tell us about the capability of nuclear power?
> This is not "doing" or "understanding" math.
Asserting something purely by stating it does not prove anything but that you intuitively believe it to be true.
Asking a random question in the middle of a discussion about something different is obviously either a troll or a bad-faith rhetorical flourish. If you actually wanted to engage, you could explain what relevance souls have to the discussion at hand, and what you expect the answer you'd receive to mean.
Often people who believe in souls, think there is something supernatural to consciousness and therefore non-humans cannot have that. If that is what someone thinks, then a lot of confusion is clarified. Those who do not believe in souls can dismiss the statement of their dismissing the question whether ai is conscious or not.
I am okay with receiving a straight answer in either direction.
Don't want to speak for Hardbass too loudly, but seems like he's asking about dualism? It's a fairly old debate, "does man need a soul to be conscious?"
+edit: I've actually been quite curious about how people might answer the soul question too, but was too afraid to ask.
I see too many comments dismissing the possibility of ai consciousness without reason or with weird reasons that seem to potentially imply a dualist thought process hidden.
Hugsbox · · focus · HN ↗
So obviously what appears right at the top is the AI summary, which told me "they've already secured their #4 position and made the playoffs". I knew this wasn't true, and I guess I could have just scrolled down a bit further and found my answer but now I was curious.
So I said "that's not true, they're still #5, what I want to know is _could they still make the playoffs_"
It says they've got an upcoming game against Ottawa, and if they win their chances are good. That game has already taken place, so I correct it again and finally I get a reasonable answer.
My question is: what's the point of the AI in the search engine if it itself isn't going to use the search engine first before answering? Like, I can't wrap my head around that. The answer is on the same page as its hallucination. It could have done a cursory look around before first hallucinating something completely false, and when corrected the first time giving me outdated information. It's meant to be A SEARCH ENGINE!
beloch · · focus · HN ↗
LLM's, in their present stage of development, are sort of like a crack-addled idiot savant. Sometimes they are obviously insane, and sometimes they seem quite cogent, but you must never trust them implicitly. This may be why they are so difficult to constrain. You could give them something equivalent to the laws of robotics, but following laws requires thought processes they simply don't have.
I'm actually sort of amazed Google doesn't make people accept some kind of butt-covering EULA and post disclaimers about the inaccuracy of results before even showing you their AI's output. Are they not being sued over this kind of thing?
VCFundedGenYer · · focus · HN ↗
walrus01 · · focus · HN ↗
But, for example, if you ask deepseek v4 flash 0731 to produce a python script to calculate the distance or azimuth directions between two points on an oblate spheroid using the vincenty and haversine geodetic formulas, it'll turn out the factually accurate vincenty and haversine formulas which has a perfect 100% correlation with what is hard coded into human-written GIS software. These things are clearly in its training data set from whatever whole-internet-crawl/scrape built the training set.
Heck, just for fun I asked a reasonably smart LLM to re-implement the Karney formula (which is considerably more complex than Vincenty), just in case I ever had a need to calculate the distance between two points down to the nanometer, and it did it: <a href="https://www.google.com/search?&q=karney+formula+geodetic+" rel="nofollow">https://www.google.com/search?&q=karney+formula+geodetic+
reference: <a href="https://github.com/pbrod/karney" rel="nofollow">https://github.com/pbrod/karney
You still have to be skeptical of its results and capable of understanding if it's gone off on a hallucinatory path, but saying LLMs can't do math isn't really a hundred percent accurate anymore. More precisely it's that they can't do the math internally but they're quite capable of producing the tool that does the math. And often producing a basic one-off tool that does the math takes less than a few seconds, then it runs it, and will spit back the results.
Deepseek v4 flash 0731 (a somewhat randomly chosen example) isn't even particularly sophisticated, large, or capable compared to a GLM5.3 size model or Kimi K3 size thing.
AdieuToLogic · · focus · HN ↗
LLMs are neither smart nor stupid. They are statistical token generators whose results are dependent upon their training data set and involve a degree of randomness.
> You still have to be skeptical of its results and capable of understanding if it's gone off on a hallucinatory path ...
Again, LLMs do not "hallucinate." They are statistical token generators whose results are dependent upon their training data set and involve a degree of randomness.
Nothing more.
See also anthropomorphism[0].
> More precisely it's that [LLMs] can't do the math internally but they're quite capable of producing the tool that does the math.
This still falls under the purvey of statistical token generation. To wit, given enough variations of:
LLMs can identify the addition expression in "What is 4 + 1?" and then emit a `'bc "4 + 1"'` command to produce a response. This is not "doing" or "understanding" math.It is pattern recognition, a task in which ANNs[1] excel.
0 - <a href="https://en.wikipedia.org/wiki/Anthropomorphism" rel="nofollow">https://en.wikipedia.org/wiki/Anthropomorphism
1 - <a href="https://en.wikipedia.org/wiki/Neural_network_(machine_learning)" rel="nofollow">https://en.wikipedia.org/wiki/Neural_network_(machine_learni...
Eisenstein · · focus · HN ↗
You haven't demonstrated why this matters.
> Nothing more.
Are you contending that complex systems cannot be more than the sum of their parts?
A market is nothing more than offers and counter offers.
A ant colony is nothing more than scent trails.
All life on earth is nothing more than reproduction with variation.
> This still falls under the purvey of statistical token generation.
Stating the mechanism does nothing to provide insight into capability. For instance: a nuclear power plant boils water by using fuel rods for heat. What does that tell us about the capability of nuclear power?
> This is not "doing" or "understanding" math.
Asserting something purely by stating it does not prove anything but that you intuitively believe it to be true.
hardbass · · focus · HN ↗
jibal · · focus · HN ↗
hardbass · · focus · HN ↗
tsimionescu · · focus · HN ↗
hardbass · · focus · HN ↗
I am okay with receiving a straight answer in either direction.
Kim_Bruning · · focus · HN ↗
+edit: I've actually been quite curious about how people might answer the soul question too, but was too afraid to ask.
hardbass · · focus · HN ↗
p_l · · focus · HN ↗