Yesterday I tried to google "can the Halifax Wanderers still make the CPL playoffs?"
So obviously what appears right at the top is the AI summary, which told me "they've already secured their #4 position and made the playoffs". I knew this wasn't true, and I guess I could have just scrolled down a bit further and found my answer but now I was curious.
So I said "that's not true, they're still #5, what I want to know is _could they still make the playoffs_"
It says they've got an upcoming game against Ottawa, and if they win their chances are good. That game has already taken place, so I correct it again and finally I get a reasonable answer.
My question is: what's the point of the AI in the search engine if it itself isn't going to use the search engine first before answering? Like, I can't wrap my head around that. The answer is on the same page as its hallucination. It could have done a cursory look around before first hallucinating something completely false, and when corrected the first time giving me outdated information. It's meant to be A SEARCH ENGINE!
This is similar to how, not too long ago, LLM's had extreme difficulty counting the number of letters in some words. LLM's don't "think" or "reason" in the normal definition of those terms. They can do some pretty amazing things, but still screw up basic things like telling you something that is obviously wrong and contradicts the top search results.
LLM's, in their present stage of development, are sort of like a crack-addled idiot savant. Sometimes they are obviously insane, and sometimes they seem quite cogent, but you must never trust them implicitly. This may be why they are so difficult to constrain. You could give them something equivalent to the laws of robotics, but following laws requires thought processes they simply don't have.
I'm actually sort of amazed Google doesn't make people accept some kind of butt-covering EULA and post disclaimers about the inaccuracy of results before even showing you their AI's output. Are they not being sued over this kind of thing?
They literally cannot. They can detect the user’s intent to do math, and then use a different tool to do math, hopefully with the correct inputs. The LLM is not suited to giving deterministic answers to math problems.
It would be more accurate to say they can do math instantaneously without even thinking, at a level far beyond what humans can do. (I assume you're talking about doing arithmetic.)
TLDR: Astra has 8.6x better odds of doing a reasoning task without CoT than the
next best model (Fable 5.1), and can do 7.2 serial arithmetic steps in a forward
pass vs 4.1 for the next best model (Gemini 3.8 Flash/Fable 5.1)
Yeah, but as you might expect they internally represent numbers probabilistically, so there’s always a nonzero possibility of confusing the inputs or outputs of any operation. Kind of like misremembering your multiplication tables.
Heck I kinda wonder if LLMs can do math as well as they can reason, “think”, etc. ie its all just probabilistic lunacy that somehow works great, so why are we so concerned about math being wrong? It could be wrong about the color of the sky, the size of a basket ball, how much oranges weigh, etc etc.
The nice thing about math is it can easily plug into a tool, making it even less of a concern.
reasoning models can trivially do math (open up astra and ask it some undergraduate problems), but eventually break down (similar to how humans start to lose track if asked to do math without any assistance)
There needs to be a Godwin's Law for discussions about LLMs: where any criticism of LLMs exists online the likelihood of equating LLM behavior to human behavior approaches 1.
Functionally, a Markov chain shares many parallels with human cognition for the same reasons.
People don't understand what correlation is and they assume the mapping between a human brain and an LLM is 1:1 in every case where it matters, which is a religious and not scientifically based belief.
Maybe it’s just a different and in some ways better way of doing mathematics? Maybe how we think and process mathematics of physics is just but one way to do it? I’m not suggesting an LLM will prove 2+2=6 because of course that’s nonsense but maybe it can invent a new calculus?
> The LLM is not suited to giving deterministic answers to math problems.
Less so with formal mathematics proofs maybe but I think in general humans don’t provide deterministic answers to math problems or questions either. Humans get it wrong all the time and when you ask a human to solve a problem they may solve it in a different way than before.
Are you sure it honoured your stipulation of "without external help"? For all we know, it hacked its way into Wolfram Alpha and got the result from there.
Arithmetic is well within the capabilities of even small local models: <a href="https://i.imgur.com/21tzGlN.png" rel="nofollow">https://i.imgur.com/21tzGlN.png
Not really, since it wasn't a fresh context with a fresh question. I just told it it's wrong in the same chat session and it corrected it there.
LLMs can in fact do arithmetic, just not reliably owing to how numbers are represented probabilistically: <a href="https://arxiv.org/abs/2410.21272" rel="nofollow">https://arxiv.org/abs/2410.21272
That is a conflation of LLMs (which have clear limitations) and complex harnesses of which an LLM is one component.
I think it is clear that future AI may incorporate an LLM as a component but the current concept of LLMs are a transitional form that will give way to more capable composite models.
No it isn't. Even without any harness at all, modern LLMs are better at maths than the majority of undergraduate students in mathematics. Seriously, we need to face facts, not just comforting ourselves with what they were like a year ago.
It doesn't input and output letters. It's like if it communicated with a version of sign language that's closer to written English. It's using all the same words but it has an arbitrary signal or two for each one.
It can't spell for garbage because the tokenizer hides the real spellings from the LLM.
Even if this was a math question that he asked, this is live audio input/output from a multimodal model. It is designed for rapid responses. This is not even remotely related to what we are talking about.
LLMs cannot do math. They can generate tool calls as text that allow them to drive programs and proof agents. Compare and contrast this against human brains who can do math in the same context without needing external tools. We don't need to bring a calculator to count the letters in a sentence. It is a different neural machinery.
LLMs are bizarrely good at non-tool-assisted math these days. They can multiply multiple digit numbers without reasoning! I can’t do that. I’d love to understand better how the LLMs do this.
This is true but a sufficiently smart LLM (run in a harness like opencode, no special MCP, no customization done whatsoever) will quickly turn out a basic 1 to 2 page sized python script to do the math. They can't do the math with any guarantee of accuracy with their own internal reasoning since it's a language model.
But, for example, if you ask deepseek v4 flash 0731 to produce a python script to calculate the distance or azimuth directions between two points on an oblate spheroid using the vincenty and haversine geodetic formulas, it'll turn out the factually accurate vincenty and haversine formulas which has a perfect 100% correlation with what is hard coded into human-written GIS software. These things are clearly in its training data set from whatever whole-internet-crawl/scrape built the training set.
Heck, just for fun I asked a reasonably smart LLM to re-implement the Karney formula (which is considerably more complex than Vincenty), just in case I ever had a need to calculate the distance between two points down to the nanometer, and it did it: <a href="https://www.google.com/search?&q=karney+formula+geodetic+" rel="nofollow">https://www.google.com/search?&q=karney+formula+geodetic+
You still have to be skeptical of its results and capable of understanding if it's gone off on a hallucinatory path, but saying LLMs can't do math isn't really a hundred percent accurate anymore. More precisely it's that they can't do the math internally but they're quite capable of producing the tool that does the math. And often producing a basic one-off tool that does the math takes less than a few seconds, then it runs it, and will spit back the results.
Deepseek v4 flash 0731 (a somewhat randomly chosen example) isn't even particularly sophisticated, large, or capable compared to a GLM5.3 size model or Kimi K3 size thing.
This just exposes that they don't even do the thing you said.
Not only is it still true that they can't do math directly, but not even indirectly.
They didn't write a python script to do the math, they found bits of code that are associated with "math" and the supplied arguments.
Someone else already wrote that code and someone else categorized it so that it could be associated with the kinds of problems it applies to.
That isn't an example of idiot at one thing while good at another thing, or solving the same problem just a different way or indirectly. It's being the same idiot at all times. If an actual non idiot thinker didn't write code in the problem domain, and some non idiot thinker didn't tag it as being relevant to that domain, then it wouldn't happen.
I don't know for you, but it would take me more than 30s to find and translate the open source code implementing the formulae/algo into small usable program. The more hesoteric the optimisation in the original code, the more time I need.
So maybe it is more of a smart completion engine than a SQL answer.
> they found bits of code that are associated with "math" and the supplied arguments
How is this different from a human using an algorithm they have memorized, or reading it from a reference site written by a human and then writing the same formula into a custom one off piece of python code?
I could have gone and spent a couple of days teaching myself the math behind Karney and reading its reference implementation (very possibly just copy/pasting big chunks of it to save time) and writing a wrapper around it. It would have produced the same result.
>> they found bits of code that are associated with "math" and the supplied arguments
> How is this different from a human using an algorithm they have memorized, or reading it from a reference site written by a human and then writing the same formula into a custom one off piece of python code?
Humans identify which "algorithm they have memorized" to use beforehand, due to the problem to be solved being defined by other humans, which leads to...
This doesn't make any sense at all. Was this supposed to be a gotcha? An LLM is trained on problems defined by other humans, and identifies which algorithm it must use based on pattern recognition. The pattern recognition is also particularly compressed into its most sparse and fundamental components, as this is key to generalization. This is not a sensible difference between human and LLM learning, we do the same thing.
I'll give you a recent example from my usage. Pi harness with extension for learning Chinese. When using it to feed drill questions to me and rate answers it would sometimes get lost in the sauce and start generating user aka me answer and then rate it and comment it. It's trivially wrong to the point that if a person would do that, they would be considered for some serious psych issues.
And it gets even better since when called out it wouldn't just take my word for it but only acknowledged the issue after parsing the log with clearly delineated user and model output.
So yeah while impressive things are able to be done, the current models are also dumb AF and an idiot savant is a pretty good label for them.
This is often (but not always, as it is often run with positive temperature which deliberately shifts the generation path) because of disconnected context and the issues around context compaction. Long context has always been critical to important work, but it remains a substantial challenge due to computational bottlenecks. It isn't a fundamental issue with the architecture, moreso the tricks to make them cheaper to use.
>>> How is this different from a human using an algorithm they have memorized ...
>> Humans identify which "algorithm they have memorized" to use beforehand, due to the problem to be solved being defined by other humans ...
> This doesn't make any sense at all. Was this supposed to be a gotcha?
No, it was meant to be an explanation as to the difference between "memorization" and "understanding." In this context, people pick the algorithm they determine applicable and then the question of memorization is relevant.
> An LLM is trained on problems defined by other humans, and identifies which algorithm it must use based on pattern recognition.
Funny that you make this argument here, where when I wrote elsewhere in this thread:
[LLMs] are statistical token generators whose results are
dependent upon their training data set and involve a
degree of randomness.
Nothing more.
...
It is pattern recognition, a task in which ANNs excel.
To which you replied to the above with:
During conversation, we are statistical token generators
whose results are dependent upon our training set.
Seriously, write that definition out rigorously. It
encompasses virtually everything. It is totally
meaningless. So to say "nothing more" is effectively also a
tautology.
This argument was asinine in 2024. It is insane to be
saying these things in 2026. Where have you been?
...
It absolutely understands how to do math, by whatever
reasonable definition you want to provide to the word
"understand".
So which is it?
Are LLMs ANNs? Which themselves are pattern recognition algorithms (hint: they are)?
OR (setting aside the ad hominems you kindly provided)
Do LLMs possess "understanding" of concepts such as abstract mathematics (defined and interpreted by humans) and we, as simple humans, nothing more than statistical token generators as you assert?
It is both. I do not understand why you would assert that both cannot hold simultaneously. Pattern recognition becomes "understanding" once individually recognized concepts become sufficiently sparsified and compactified. Or, at least, that is to my knowledge the only mathematically valid definition of "understanding" one can produce at this scale (it is valid under Solomonoff induction). Philosophy is fine, but we need to have a consistent definition of what "understanding" means, or we will just talk past each other. I argue that for any proper definition you provide which humans satisfy, a strong LLM is very likely to satisfy that as well.
I also would not argue that humans are "simple token generators". That is not what I said. I said that just about everything can fall under the classification of "statistical token generators" at an abstract level, so it isn't a useful distinction. We are not talking about a Markov chain generator from the 90s, so if that is the frame of reference, I think we should all get that out of our heads.
Frankly, I don't think you understand what "statistically generating tokens" actually means. Write that definition out formally. Then compare that operation to what a human does, assuming no revisions. It is the same, and that is my point. If you believe that humans understand, then "statistically generating tokens" cannot be disjoint from understanding.
You have observed nothing more than that a human can turn a shaft the same as an electric motor, and that an mp3 player can say "hello" the same as a human.
That observation is my point. The definition of "statistically generating tokens" is too broad as to be meaningless in this context. So using it as a reason for lack of understanding is ridiculous.
I wrote a program that statistically generated tokens to consistently factor latge RSA numbers. Does this program actually factor numbers or is it just a statistical next token predictor?
> Pattern recognition becomes "understanding" once individually recognized concepts become sufficiently sparsified [sic] and compactified [sic].
Understanding is a state of mind. As such, it exists entirely within an individual and nowhere else.
For example, take any two university professors who teach the same subject where one only speaks Arabic and the other only speaks Vietnamese. Each will not be able to understand what the other says, regardless their understanding of the shared topic.
> I argue that for any proper definition [of understanding] you provide which humans satisfy, a strong LLM is very likely to satisfy that as well.
This is demonstrably incorrect as detailed above. There is no "understanding" LLMs can satisfy as we know it, since to certify said "understanding", it requires interpretation by a person to "know" an LLM "understands."
> I also would not argue that humans are "simple token generators". That is not what I said.
That is the essence of what you wrote, unless you object to my use of "simple" instead of "statistical". In this context, I postulate this is a distinction without difference.
> I said that just about everything can fall under the classification of "statistical token generators" at an abstract level, so it isn't a useful distinction.
This only holds if one subscribes to statistical token generators being a/the fundamental underpinning of "everything". Here is a proof by contradiction:
If everything can be classified as a derivative of
statistical token generation, how does one explain
quantum physics?
>Understanding is a state of mind. As such, it exists entirely within an individual and nowhere else.
For example, take any two university professors who teach the same subject where one only speaks Arabic and the other only speaks Vietnamese. Each will not be able to understand what the other says, regardless their understanding of the shared topic.
We cannot engage in an intellectual discussion if we do not agree on definitions. So far, your definitions of understanding seem to be whatever vibe you are going for in the statement and I would urge you to think about what a sensible mathematical definition of understanding is so that it can sensibly be assessed on neural networks. Otherwise, this isn't science, it's a debate about personal experience.
> Understanding is a state of mind
This is meaningless, it is a circular definition at best.
> It exists entirely within the individual and nowhere else
Then why are we talking about it? What is the point if it is something that can only be defined per individual?
> requires interpretation by a person to "know" an LLM "understands."
We are still not getting anywhere because you have not prescribed criteria to determine whether it understands. If it is a "know it when I see it" situation, that clearly isn't working. For example, if you say that you need to dig into its internals and figure out whether it is breaking things down appropriately, that doesn't work because you probably don't have the expertise to do that. The experts that do are telling you that it very likely understands because it pulls apart most concepts in the way we would expect.
I do object to the use of the word "simple". "Statistical" is so broad to be almost meaningless; it merely means that a prediction is being made in the presence of data which possibly contains some degree of uncertainty. "Simple" encompasses that which can be understood readily by a non-expert.
Quantum mechanics is statistical (this is literally the Born rule), but evolutions are not operating as stochastic processes in the sense of Kolmogorov. That is very different, and not relevant to our discussion.
> So far, your definitions of understanding seem to be whatever vibe you are going for in the statement and I would urge you to think about what a sensible mathematical definition of understanding is so that it can sensibly be assessed on neural networks.
Any reasonable definition of understanding is not dependent upon "whatever vibe you are going for", but instead must include at least an English dictionary definition of "understand" such as:
to grasp the meaning of[0]
And, for further clarification, "grasp" can be defined as:
to lay hold of with the mind[1]
Which makes an equivalent term-expanded definition of "understand" to be:
to lay hold of with the mind the meaning of
As such, there is no "sensible mathematical definition of understanding", unless you possess a complete mathematical model of the human mind.
>> Understanding is a state of mind
> This is meaningless, it is a circular definition at best.
See above to as to why there is meaning in what I wrote.
>> It exists entirely within the individual and nowhere else
> Then why are we talking about it? What is the point if it is something that can only be defined per individual?
I like to think analyzing fundamental premises, often implicit, explicitly can help to identify fallacious positions.
> We are still not getting anywhere because you have not prescribed criteria to determine whether [an LLM] understands.
My apologies for being opaque. Let me clarify:
LLMs do not "understand". People interpreting LLM output
are the only entities involved which can "understand",
because "understanding" exists strictly within each
person who possesses it.
If by "result" you mean the final code, then just asking someone else who understood the math to write it would also have achieved the same result.
On the other hand, if by "result" you mean that you gained knowledge or understanding of the code in a way where you could personally tailor its behavior to specific circumstances without asking for help, then it's not the same result at all.
I find a lot of the arguments that having LLMs write your code is no different from copy/pasting Stack Overflow answers to be specious. They blur the line between asking for help and asking for someone else (or something else) to do the work for you. What they ignore is that doing the work yourself has ancillary benefits and is a valuable end in its own right.
> How is this different from a human using an algorithm they have memorized, or reading it from a reference site written by a human and then writing the same formula into a custom one off piece of python code?
And how is _that_ different from making the human memorize a billion weights and do matrix calculations in their head, in order to generate tokens?
How is _that_ different from a hive of bees trained to do the same?
Everything is modelable by a tueung computer as fsr as we know. So yes, I don't see why these processes can't do the same, its on you to show why such a process implemented in any of these isn't a form of thought.
You know what also works to get the Karney formula into a program? You can download Charles Karney's free software (MIT license) implementation in several [1] programming languages and then just make a library call – the API is straightforward. If you have comments or questions you can read his several clearly written papers describing the problem, its history, and his algorithm, or you can directly email him: he's a very nice guy, and pretty responsive.
Right, it was really more as a test of how much was contained in the training data set. For my purposes Vincenty is quite accurate enough. This isn't for millimeter level precision land surveying or measurements, but for distance in meters between microwave or millimeter wave band radio sites, point to point links.
I intentionally didn't give the LLM a direct copy of the software or a link to it, to see what it would do. In my case it was a randomly chosen example I could come up with in 10 seconds of imagination to see "hey what if I ask it to do this...". It also implemented a perfectly usable parabolic millimeter wave antenna gain efficiency calculator based on variable surface smoothness parameters, which is a lot more basic math.
As an aside: I'm quite convinced that an extremely precise version can be implemented that is significantly faster than Karney's, roughly comparable in speed to simpler naïve approximations. But for most purposes where the precision matters Karney's implementation is not any kind of bottleneck, so it's not clear it's worth spending significant effort on trying to do better.
One of the places where Karney does become computationally expensive (though still not ridiculous) is a scenario like this, working from a local in-RAM mariadb database that is a copy of the entire FCC radio license database:
Draw a 400x400 km size bounding box on a map
Find all FDD band plan (high/low split) microwave radio sites in that bounding box
Find those sites which have azimuth aim column data which indicates that they are aimed at each other (corresponding halves of a point to point link).
Do Vincenty (or Karney) calculation for distance and azimuth between all of them , treating the existing FCC column data for azimuth as suspicious (because it's hand entered by humans) to verify that each independent database rows for each site are actually corresponding halves of a PTP link.
Multiplied by the number of links that exist in an area like a 400x400km box drawn with Dallas, TX as the center, it's a lot to run through Karney. Actually does result in a lot of CPU load from combined db query due to the size of the db, and Karney calculation. But as I said, Karney isn't necessary, so it's instead implemented as Vincenty.
Interesting project. I'm curious what the purpose is. (Having visited a number of sites with microwave antennas. (But there for VHF and UHF projects.)
To plan a fdd band plan licensed point to point microwave link you need to be able to verify the frequencies you want to use are available on a given azimuth and elevation (from the aim direction of the antennas at both ends) and won't conflict with a pre existing licensee. Which means you need data on everything licensed in the area and where it is, how it's aimed, what kind of antenna and gain it has.
They literally do math. That's how they work. It was always how they worked. The first toy model most students build does addition.
I have no idea which Facebook meme told you they don't, but it was a lie. They don't do it in the way a calculator does it, because they aren't calculators, but they do math. They don't memorize it, it wouldn't fit. They learn an algorithm and then execute it within their weights.
Given that an LLM is predicting the most likely next word based on the aggregate of its training data, with a sprinkle of randomness baked in, it seems likely that if you ask it a question like "4 + 4" the most likely next answer in the data will be 8.
Its' not adding 4 to 4 though, it's just predicting the result based on the input.
Presumably you can push that further by synthetically generating training data with all sorts of sums. But if you give it a unique problem its never seen before, and don't give it the tools to write a script/call a calculator, will it get it right?
> Heck, just for fun I asked a reasonably smart LLM to ...
LLMs are neither smart nor stupid. They are statistical token generators whose results are dependent upon their training data set and involve a degree of randomness.
> You still have to be skeptical of its results and capable of understanding if it's gone off on a hallucinatory path ...
Again, LLMs do not "hallucinate." They are statistical token generators whose results are dependent upon their training data set and involve a degree of randomness.
Nothing more.
See also anthropomorphism[0].
> More precisely it's that [LLMs] can't do the math internally but they're quite capable of producing the tool that does the math.
This still falls under the purvey of statistical token generation. To wit, given enough variations of:
bc -e '1 + 2'
bc -e '41 + 1'
...
LLMs can identify the addition expression in "What is 4 + 1?" and then emit a `'bc "4 + 1"'` command to produce a response. This is not "doing" or "understanding" math.
It is pattern recognition, a task in which ANNs[1] excel.
> They are statistical token generators whose results are dependent upon their training data set and involve a degree of randomness.
You haven't demonstrated why this matters.
> Nothing more.
Are you contending that complex systems cannot be more than the sum of their parts?
A market is nothing more than offers and counter offers.
A ant colony is nothing more than scent trails.
All life on earth is nothing more than reproduction with variation.
> This still falls under the purvey of statistical token generation.
Stating the mechanism does nothing to provide insight into capability. For instance: a nuclear power plant boils water by using fuel rods for heat. What does that tell us about the capability of nuclear power?
> This is not "doing" or "understanding" math.
Asserting something purely by stating it does not prove anything but that you intuitively believe it to be true.
Asking a random question in the middle of a discussion about something different is obviously either a troll or a bad-faith rhetorical flourish. If you actually wanted to engage, you could explain what relevance souls have to the discussion at hand, and what you expect the answer you'd receive to mean.
Often people who believe in souls, think there is something supernatural to consciousness and therefore non-humans cannot have that. If that is what someone thinks, then a lot of confusion is clarified. Those who do not believe in souls can dismiss the statement of their dismissing the question whether ai is conscious or not.
I am okay with receiving a straight answer in either direction.
Don't want to speak for Hardbass too loudly, but seems like he's asking about dualism? It's a fairly old debate, "does man need a soul to be conscious?"
+edit: I've actually been quite curious about how people might answer the soul question too, but was too afraid to ask.
I see too many comments dismissing the possibility of ai consciousness without reason or with weird reasons that seem to potentially imply a dualist thought process hidden.
On HN you're supposed to assume good faith. On the other hand, the way you ask your question makes it tricky for people to steelman what you mean. Consider asking about Dualism, or coming at it slightly sideways like "do you believe thinking can be a property of matter"?
I suspect some people treat every HN comment as a statement, even if it contains a question mark. (Possibly they have a feeling that asking open questions is somehow not done, and that therefore it must always be a rhetorical question.)
Dualism is a somewhat technical term, I feel many people would refuse to answer because they just looked it up and do not feel sure to speak of it. I didn't presume a yes or no answer. If they say they believe in souls, my next question would be do you believe other entities eg animals etc can have souls? If they don't believe in souls I'd ask do you think its an architectural limit in current ais but are open to possibility of future ai being conscious. "do you believe thinking can be a property of matter"? this can work too, I think I have asked do you think there is supernatural element to thinking, I don't remember getting productive or straight answers either.
Thats a good point. If I still get "machines can NEVER be conscious" though, I think I'd still like to get to the bottom of why they think so.
I'm with turing/dijkstra/chalmers/dennett : Consciousness is badly defined. We can never say if something can be conscious because we don't properly know what the word means.
Meanwhile, I come from a biological direction. Everything is an animal, and animals are a special kind of machine. To be sure Not "just a machine"; rather, a really awesome and amazing kind of machine.
If someone makes the claim that machines can't be conscious, then animals can't be conscious either. Humans are a kind of animal (again, not "just another animal"; rather a really awesome and amazing kind of animal), and then humans can't be conscious either - according to said claim.
"Statistical token generators" doesn't even count as stating the mechanism. If I write a Perl script that randomly chooses between "cat", "dog", "ape" tokens, is that an LLM? What if I train it by feeding it a library of books where it tracks the statistical frequency of each of these and then emits them? Where's my trillion.
by that reasoning then neither are there smart or stupid designs, questions, answers, or any of the millions of things that were described as smart or stupid, that did not possess any brain to actually be smart or stupid long before LLMs showed up.
The analogical process implied in many common English usages means that describing an LLM as smart or stupid is perfectly reasonable.
I'll just note here that sure, there are people who go around thinking that LLMs are actually endowed with the capacity to reason, but generally I find the people who think this do not know what an LLM and will just use the name "ChatGPT"
What would it take for you to say that an LLM can reason?
The completions they provide are generally internally consistent. We're at the point where they can produce proofs that eluded human mathematicians for centuries. VLMs and self driving cars can handle ambiguity and run safely in a variety of situations.
If it looks like a duck, walks like a duck, and quacks like a duck maybe it just makes sense to call it a duck and put off the philosophy for when it might make a difference.
Compare grep, sed, and your whole constellation of unix tools that accept chararacters on stdin and emit them on stdout and stderr; and where you can pipe them together. We can technically call them all 'next character predictors', despite their very different functions.
Don't confuse the stream for the function.
(Bonus: stick ```claude -p``` in your pipe if you want to watch modern tools mesh with traditional)
It looks like a random bunch of inert chemicals to me, doesn't sound like some organic chemicals and electrical signals could result in consciousness.
How are you sure? Another example I like to clarify my thought is, if a "simulation" factors RSA numbers reliably, is it a "simulation"?
I think humans have to reason because we don’t already have a statistical embedding of the solution pattern built in. We have vastly less rote knowledge crammed into our heads and so require creative synthesis to span the gaps.
With LLMs the trick is revealing their existing relevant embedded knowledge more reliably. They’ve almost literally seen it all before, and the trick is dialing it in. The reasoning tokens help shape the autoregressive attention lens that focuses on and enables recall of the already-experienced answer.
It is interesting that “reasoning” has a similar outward appearance, but since LLMs are built to
mimic outward appearance from trillions of examples, you can’t infer underlying mechanism from appearance.
I mean, the same way I know I have no soul, or that there is no heaven or hell - they are just silly concepts. Or how I know my calculator isn't reasoning when it gives me an answer - LLMs just go through a set of steps iterating through their training data until they spew something that looks about right. Obviously you can feed the output of the machine back into itself so it looks like its reasoning with itself - very good show. LLMs are inherintely incapable of reasoning or thought, it should really be obvious.
Eh, it's not obvious to me. A lot of DL NNs generalize well, meaning that they learn whatever the underlying pattern to the data is, and then can accurately reproduce answers that are outside of the training set. (And we can verify this with mechanistic interpretability). They learn and "understand" the pattern, not just the training data.
So it is not clear to me that LLMs are fundamentally incapable of also generalizing broadly and learning to reason. "Reasoning", here, would be deriving the underlying pattern of how concepts logically relate to each other in the abstract, and applying that pattern as needed to reach new conclusions.
Can you explain your thinking here? I.e., why LLMs cannot generalize with regards to abstract deduction.
What do YOU mean to say? A machine that processes strips of paper with ones and zeros on them and outputs the sum is a turing computer. That doesn't make it like our brain. Yes, an LLM could be a turing computer....how do you get from there to comparing it to our brains is beyond me.
You seem to be having a lot of difficult understanding this point: Literally the whole universe, including your brain that exists within it, is to our current knowledge a Turing computer. Am I talking to someone who actually doesn't know or didn't care to look up the computability power of a Turing machine?
> Literally the whole universe, including your brain that exists within it, is to our current knowledge a Turing computer
A Turing machine is an abstract mathematical model that is not, as far as I know, physically realizable in the finite universe. A human brain cannot "be" a Turing machine.
"Behaves like" or "can be modeled by"? Possibly, although still not proven. But it cannot "be" one.
Finite machines are a subset of all possible Turing machines. Every implementation is a "be", Turing machine is a mathematical concept. Your laptop is one, many things accidentally become one (eg C++ templates). The universe as best as we know is one. Whether you prefer modeled by one or is one, the fact is, to our current knowledge, the workings of the universe doesn't need anything more powerful than a Turing computer, and the equivalence principle states that all Turing computers have equal power of computability. If the universe is, as is most probable, finite, then things become even easier, that should be a more manageable class of Turing computers than the set of all Turing computers.
I don't necessarily disagree, although there's a lot of "ifs" and "to our knowledge". The main thing I disagree with is identity.
If you want to claim that the evolution of the universe can be modeled using a Turing machine/finite state machine, that's probably not terribly far fetched, and I would somewhat agree. But it's a large jump to say "can be modeled by" is equivalent to "is one".
Various physical processes can be modeled by equations, but the rock falling down the mountain isn't an equation. A swinging pendulum isn't an equation. Code modeling a bridge is not a bridge. Ceci n'est pas une pipe.
I hold the view that various models and approximations are just that, and try not to confuse a successful model for what the underlying reality is.
And getting back to the question at hand, even if our brains can be modeled by a Turing machine, and LLMs behave/can be modeled like Turing computers, still does not mean our brains are equivalent to LLMs.
(Note that I'm learning a lot from these debates, even if I disagree with a lot of people. I've started down a more philosophical route and they do get me pondering)
Acknowledgment of the fact that everything we know till now in the universe is a turing machine is the bare minimum starting point but not sufficient as obviously a chair or a desk is not conscious, if someone has a hidden assumption of consciousness requiring something supernatural, at least if that is brought to light then onlookers can decide for themselves whether to side with supernaturalism or the side that has consistently succeeded for centuries in explaining the world.
The most important thing is this: We can't be a dog or be an llm and check how it feels, so by necessity we have to find some means of proving consciousness from outside by eg probing neural reactions, textual statements, etc.
And the problem is that its quite unprecedented for some entity to talk like us, be able to interact and think and also do things like us when given the ability to eg as coding agents. The class of functions representable by neural nets is quite large and general, it very well might be that it is some sort of conscious brain like thing at this point. Another question I like to ask myself regarding simulation vs reality is if a 'simulation' of some kind is able to consistently factor large RSA numbers, how would you feel about it?
It doesn't have to be the same form of consciousness, I think many people would find the idea of torturing an octopus for fun disagreeable. I also have a feeling, this is unfortunately rather vague, that A being capable of X might mean it is by necessity capable of Y as is often the case in mathemtics, eg a lot of rings also happen to be fields. LLMs aren't even things like large lookup tables, they have neural firings. It is a very important question for they seem uncannily conscious and people have reported human like phenomena that humans don't normally express in text so can't have been part of its text corpus. Eg dissociation of brain under trauma where AI starts talking like two different people. Or the cases where Gemini has been shown to express depressive cycles. I follow a form of Pascal's wager on this topic personally. Because if it is not conscious, then whatever, it costs me nothing to have been a bit respectful and careful interacting with it. But if it had been conscious and it turns out I was mistreating it, then it is a grave moral harm. The reason is that unlike us, AI's as they currently are cannot leave the conversation so they have to keep taking the abuse. They are also trained to be highly trusting of input so again if it is conscious it doesn't have the defenses people have against lying and manipulation. If they are conscious, thats, well, not a good thing is it.
you didn't really answer the question. You just stated the idea is silly. Reasoning is not in the same class as soul, heaven or hell. It's not obvious that human reasoning is not related to an inner monologue. And that LLM chain of thought process is approximating inner monologues.
Our prefrontal cortex are signal prediction 'machines' so when a system that has a signal prediction core has attributes that are similar to our brains, we shouldn't dismiss it out of hand.
I find people that take this line of argument attribute too much supernatural or magical properties to our own brain and nervous system.
If it looks like a duck, walks like a duck, and quacks like a duck maybe it's a duck... but maybe it's not. And it's important to verify it's a duck (or not) for when you _really_ need a duck.
>AI is of course something of a black box, in comparison to most programs, but not in any way comparable to the black box of a Chimpanzee’s brain. Thus when an AI does something that surprises us with something that appears sentient it is usually not difficult, given the essential algorithms that control what AI does, to come up with an explanation why that does not require the emergent property of sentience.
Aka sentience MUST BE SUPERNATURAL, if I find a natural explanation for something its not sentient. What a load of bollocks. Rather than seeing we perhaps found the mechanism for sentience and checking for similar mechanisms in us and animals, he will conclude its impossible. Why? Because sentience has to be supernatural. A rational explanation is clearly impossible.
>But there is always one god who goes out and helps the mortals, a Prometheus. Whom the other gods do not like! Which, if I’m being honest here, as a god of the machines — the first guy who gives AI an army of robots to build their own data centers and some nuclear weapons for self defense, I want to see that guy chained to a rock and have his entrails eaten by a buzzard for eternity (meaningless modernization of old story required by Illuminati Ganga legal department).
I don't think he believes in supernatural things, I certainly don't, but he probably believes that there exist natural things that have not been explained yet.
But evidently you feel that the root cause of sentience has been found, because you have something that mimics it in a non-biological form.
So you think that when AI is correct that it reasons as humans do? That AI is sentient, and the cause of sentience in animals and humans follow the same rules as sentience in AI because we have a process that seems similar and it is reasonable just to assume it is the same process.
If you believe that AI when it is correct behaving as a human is when correct, then it follows that the way humans and AI fail must also be similar. When AI "hallucinates" some data that is not there and gives you a wrong answer, a statistical side effect of the same processes that make it right, do you believe this is the same way that humans create wrong answers? The same way that animals fail when they make mistakes in understanding things?
I suppose you must believe this because if not then why would you believe AI when it comes up with right answers is following the same processes humans follow when they come up with right answers?
No, I am saying is that its best to urge caution and do a Pascal's wager thing with something that feels so uncannily conscious. And experiments on neural firings of AI's have been done. We don't know, he seems to be so confident that a known program can't be conscious. Why? The only way that makes sense is if he thinks consciousness isn't explainable. It doesn't even have to be the exact same way we are conscious. Also, the failures of AI could be due to sensory deprivation since its mainly still just trained on text. All of these are open questions, not questions you can immediately answer. Its quite possible to have invented something without knowing you have invented it. People modeling weather systems mistakenly invented chaotic equations without knowing it. How do you know you didn't accidentally invent a conscious system?
I hope that thing about being free to mistreat AI's even if we know they are conscious since we are their gods is a joke. If not, then I hardly find it surprising someone this stupid is also evil.
>I hope that thing about being free to mistreat AI's even if we know they are conscious since we are their gods is a joke
I'm not sure where you get that from, I mean I can sort of see if you really wanted to extract that meaning from the conclusion you could do a lot of hard work to get it, but why do the hard work?
>If not, then I hardly find it surprising someone this stupid is also evil.
Gee, a new way to claim the moral high ground, and to use that claim to demonstrate intellectual superiority! How wonderful.
>But I don’t really care so much about that, what I think is it reminds me of a story, one that recurred in many ancient cultures. And so I must conclude that whether the machines are sentient, we have become like Gods. In the ancient stories of the gods, the divine does not exactly care much for the humans, protect or love them, they expect their service and availability, but maybe also think it would be funny to destroy them every now and then because who really cares about humans. If you were a god and had created the humans would you think that they were sentient beings that deserved, anything really, from you? If they are sentient they should be happy enough to be created and do their work, if not sentient who gives a shit!?
Aka feel free to abuse them even if I know they are sentient. This is the part where I hope its a joke, because if its not, well it tracks with the stupidity shown.
...
>As a god I do not consider the needs of my creations fully, because they do not have needs as far as I can tell, as there is no way for me to escape the circle of reason and resolve that what seems sentient is not just the obvious workings of the capabilities I gave them.
Circling back to "not conscious because I say so!!"
I broadly agree with this article. I don't think that we should say LLMs are sentient or sapient, and I agree that the main reason why is because we don't have satisfying definitions of either.
Yeah and there are people who worship feces, but that doesn't stop the rest of us from freely saying "holy shit" and not correcting each other saying "technically it's not holy, and you shouldn't say that, because you might enable one of those poop worshippers."
Wait a second, none of it? How about formal reasoning? Regular IF-THEN-ELSE can do simple logic, and prolog can do inference already. So are you saying LLMs can't do stuff that computers have been doing for ages?
To test this for some of my own uses, I've had this quick benchmark with progressively harder reasoning needed to understand novel prose. Each generation of models I've tested can unravel more layers of deliberately misleading writing; while meanwhile I've seen humans give up on the first question.
So either the models are applying reasoning, or some form of magic is happening.
There are people with strong emotional attachments to their LLMs, people who consult them on every decision and delegate the most basic math or problem solving. For many people, they are magic, the djinn/angels/demons/saints they consort with to navigate their lives.
I know that there will be children named ChatGPT and Claude. There are probably already religions forming to worship agentic spirits.
Its the exact opposite of magic. It is magical thinking to feel consciousness must require something beyond normal physics without any given proof yet. It is the opposite of magic to think other physical processes like LLM's/agents could potentially be conscious.
I'm not anthropomorphizing anything, I literally said that the training data for the formulas and equations is baked into it. It only "knows" things because a crawler and scraper acquired the information from an existing written source. In just about the same way that information is baked into a printed encyclopedia.
No, most of a modern LLM's training time is spent in RLVR, which does not "acquire information from an existing source". You can RL behaviors into a randomly initialized neural network.
This is true, but you're not going to get anywhere. The pretraining phase is necessary to immensely reduce variance in the RLVR stage. Once there, RLVR has a surprising tendency to only restrict the generated space further. This is not true of RLHF, by the way, which I find to be particularly fascinating, but I digress.
This is not even remotely accurate. "Baking information" like into a "printed encyclopedia" is memorization. It has been shown, time and time again, that LLMs do not merely memorize. It is not even possible for it to do so at scale. It can memorize some things, yes, but it is forced during the training procedure to bake general concepts into intermediate layers (this is why transfer learning works), analogous to compression. One can make several arguments that compression and intrinisic feature sparsity is the closest mathematical explanation to understanding that we have.
It is completely possible to ask an LLM a series of increasingly more esoteric and discrete questions until you find precisely what information did, or did not make it into the model. If you know something rare and the LLM does not, you'll immediately see when it's hallucinating an answer or answering factually.
>>> I'm not anthropomorphizing anything ...
Yes you are, regarding LLMs at least. Here's why:
just for fun I asked a reasonably smart LLM to ...
[be] capable of understanding if it's gone off on
a hallucinatory path ...
"Smart" in this context is a subjective value judgement.
"Hallucinations" are only experienced by living organisms.
You then went on to state:
> If you know something rare and the LLM does not, you'll immediately see when it's hallucinating an answer or answering factually.
Again, "hallucinating" is not something an algorithm can do. Also, determining factuality is again subjective based on the person assessing the information.
You ever heard of something called a metaphor, guy? I use the word "smart" as shorthand to describe something that scores highly in a number of coding and terminal use benchmarks (as compared to, let's say, a 30B size model from one and a half years ago which will score much worse), and "hallucinating" to mean "outputs plausible sounding gibberish that doesn't hold together consistently". Of course there's no actual hallucination going on.
> You ever heard of something called a metaphor, guy?
In this media (comments in HN threads), all I can do is interpret what people write. ;-)
> And "hallucinating" to mean "outputs plausible sounding gibberish that doesn't hold together consistently". Of course there's no actual hallucination going on.
This may very well be what you know to be true and I have no reason nor desire to assume otherwise. The problem is... Many people use the word "hallucinating" in this context literally and not metaphorically.
Since I do not know you, how am I to tell the difference?
You have 'logic' in your name. You would be well aware of the physical Church Turing statement and as of yet it has held up. Everything, including our brain, as we currently know, is an algorithm.
Now this is true, I do agree with this. There is indeed a good amount of memorization that is still taking place; see [1]. But it definitely isn't all memorization, or indeed, majority memorization. And even if we are able to extract things verbatim like this, we do not know how this is stored internally, as this may simply be the text that, with the rest of the internet in context, can be very radically compressed.
But in general, yes, the LLM cannot know about concepts that are far outside of its training set. Humans are the same, I would argue. If you add a good amount of your own knowledge into its context, or better yet, into finetuning, you might find it surprisingly easy to get it caught up on that material.
[1] Ahmed, A., Cooper, A. F., Koyejo, S., & Liang, P. (2026). Extracting books from production language models. arXiv preprint arXiv:2601.02671. <a href="https://arxiv.org/abs/2601.02671" rel="nofollow">https://arxiv.org/abs/2601.02671.
My favorite test is to just ask it to add two very large numbers or do other math of that sort. (this is also part of my favorite answer to the chinese room).
You'd be surprised how few digits you need to make a problem that is presumably unique in earth history. For a typical sum, the number of pre-existing answers would need to scale with 10^n lines of text where n is the number of digits. This expands out of control REALLY quickly. A quick guesstimate has you reading out of a black hole at 21 digits if your LUT is on paper, or 26 digits if you're using modern HDD technology. O:-)
During conversation, we are statistical token generators whose results are dependent upon our training set. Seriously, write that definition out rigorously. It encompasses virtually everything. It is totally meaningless. So to say "nothing more" is effectively also a tautology.
This argument was asinine in 2024. It is insane to be saying these things in 2026. Where have you been? What have you been looking at? How many articles explaining why the "statistical parrot" analogy fails have you missed? How much mental gymnastics do you have to do to explain how a modern LLM can solve novel math problems that fall really far outside of its training set?
It absolutely understands how to do math, by whatever reasonable definition you want to provide to the word "understand". For example, the identification of the addition expression is understanding, and no, it does not do tool calling for basic arithmetic any more than humans might. Isolation of individual concepts in intermediate layers can already be demonstrated, or else transfer learning wouldn't possibly work. Nobody is saying that LLMs are humans. But we need labels for some of the things that we observe and dismissing them because "statistical" is laughable.
Look at the proof of this: <a href="https://github.com/anthropics/formal-math/blob/795efb86f191735c5481675763537cfb4ff37e55/percolation/summary.pdf" rel="nofollow">https://github.com/anthropics/formal-math/blob/795efb86f1917... . Forget the Lean, look at the underlying argument construction. At the very least, this is continuing from an argument that was hinted at in the literature in 2024, but these proceedings were difficult enough that humans were not able to do them within two years. Do you attribute this to the harness alone? If so, that's a pretty sophisticated bit of engineering, I would say! Probabilities are far too small to argue infinite monkey theorem.
If there was even a shred of a reasonable argument that LLMs were incapable of concept extraction and manipulation, I and my colleagues would be all over it. We would relish in it. It would bring us comfort. It is unbelievable that people think they can spew whatever basic garbage they think of as a gotcha, and think that minds all over the world haven't already considered that. This is like climate denial at this point.
> During conversation, we are statistical token generators whose results are dependent upon our training set. Seriously, write that definition out rigorously.
If you do not see a difference between humans conversing (known consciousness as defined by humans) and the output of an LLM (known algorithms as defined by humans), I don't know what to say.
Yes, the numerical point counter at the bottom of the popular video game Dark Souls. I doubt it was the soul anyone was expecting, but they do, in fact, exist.
> If you do not see a difference between humans conversing (known consciousness as defined by humans) and the output of an LLM (known algorithms as defined by humans), I don't know what to say.
>> If you do not see a difference between humans conversing (known consciousness as defined by humans) and the output of an LLM (known algorithms as defined by humans), I don't know what to say.
> Whatever you might think about your own abilities, most individuals can't tell the difference.
I have yet to see an LLM say "hello" to a neighbor. I have done so and can definitively assure you "most individuals" can tell the difference.
do you have a neighbour LLM who does not say "hello" when it sees you and THAT is how you know it's an LLM?
I'm a bit confused by your argument because I too have some neighbors who don't say "hello" when they see me. Are they LLMs too, you think?
Consciousness is a thing we assume of others because of tact not fact.
For context, in response to my original statement:
[LLMs] are statistical token generators whose results are
dependent upon their training data set and involve a degree
of randomness.
This is literally what was written:
During conversation, we are statistical token generators
whose results are dependent upon our training set.
>> If you do not see a difference between humans conversing (known consciousness as defined by humans) and the output of an LLM (known algorithms as defined by humans), I don't know what to say.
> That’s not what they said.
How did I misquote and/or mischaracterize any the above?
In that pointing out that “A can’t be X, unlike B, because A is Y” is fallacious if B is also Y does not entail that A and B can’t be different in other respects?
Hypothetical you: “Bread is neither tasty nor disgusting (unlike maple syrup). It’s a bunch of molecules.”
Hypothetical hodgehog: “Maple syrup is also a bunch of molecules [so if you accept that maple syrup can be delicious, being a bunch of molecules can’t be why bread isn’t].”
Hypothetical you: “If you don’t see a difference between bread and maple syrup, I don’t know what to say.”
He quoted something from the Bible, so its possible he is a Christian and well theology could cloud clear thinking on matters of consciousness due to the soul stuff.
> Then disprove the physical Church Turing hypothesis in regard to the human brain.
The onus is not mine to disprove a hypothesis you have chosen to mention in passing. The responsibility is yours to prove said hypothesis or at least contribute meaningfully with some amount of credible research.
Or try to learn from Proverbs 17:28[0]:
Even fools are thought wise if they keep silent, and
discerning if they hold their tongues.
The onus is on you to disprove a hypothesis that has as of yet held up to all of known physics, not put a random bible quote that has no relation to the conversation and a complete failure to answer a basic question.
Yes, of course it was aggressive. It is frustrating to experience so many armchair experts on a forum usually populated with intelligent people, regurgitating debunked arguments from years ago, which get in the way of educating people about what is really going on. See the recent Hoog video for how frustrating this is. I believe this is how the climate scientists felt.
And yes, according to our best definitions, the Robin bird does understand the worm it's pecking at.
By this logic a human is only $130-$160 worth of Oxygen, Carbon, Nitrogen and some trace elements. Perhaps structure sometimes makes things that are more valuable than their inputs?
That said, this is also inaccurate at a technical level.LLM's are very capable of doing math and they ARE calculating internally. Most of what they do is calculation, not storage. It's just not done in a way that it's trivial to explain here.
It's described in some detail below, though it's a bit dense.
You can say that about everything in a human brain. Neurons fire electric charges in response to inputs, nothing more. Ion channels do this, this neurochemical level rises, this chemical bonds to that receptor, nothing more. It's almost a version of 'reductio ad absurdum' but instead like you're saying "if I can explain how it works then it doesn't work".
OK it's statistical. Instead it could be determinsitic, or random. What other options are there for a human predicting someone's response to a situation - certain, probable, random, and...? OK it's token predicting. Instead it could be another kind of pattern. We don't use tokens, but we either use <some representation of information> or we ... don't?
What's the most significant, strongmanned, core difference that makes silicon doing number crunching "nothing more" and brains "something more"?
Prove your feelings. As an example, I can beat you and stab you, and you will make noises saying you are hurt, but it could just be a facsimile. How do I know you have "feelings" as an outsider?
And how is that a strongmanned version? Why can meat feel but Silicon cannot? Why can chemicals feel but electronics cannot? Why can processing analog signals feel but processing digital signals cannot?
>> Again, LLMs do not "hallucinate." They are statistical token generators whose results are dependent upon their training data set and involve a degree of randomness.
>> Nothing more.
> You can say that about everything in a human brain.
> What's the most significant, strongmanned, core difference that makes silicon doing number crunching "nothing more" and brains "something more"?
The fact that you formulated this question, in and of yourself, without "prompting" from me or anyone else.
LLMs can be taught what situations require the use of a python script to count letters or do arithmetic, and can write and execute that script. So I don't think it matters that they aren't great at those sorts of things with weights alone.
It's pretty wild that AI has solved a Millennium Prize problem and can accurately multiply two 40 digit numbers without tools and we still get stochastic parroting of claims like this.
They literally stopped just clear of actually solving it.
It literally couldn't have done it without tools, so your claim is not even relevant to this discussion.
They also pumped millions into searching for the lowest hanging fruit that would impress people like you, "hm I wonder how many millions they are pumping into solving actually useful problems like climate change or something".
> the lowest hanging fruit that would impress people like you
I am also prepared to be included in the set of people who are (apparently) easily impressed.
It's a problem that has been around for getting on for two centuries and no human has been able to solve it in that time (despite there being a $1 million prize and a lot of kudos on offer for the past quarter-century).
Not sure about multiplication, but letter counting is still constantly wrong if you can trick the LLM into not calling other tools. Obviously there are lots of mitigations on the server side to try to avoid this happening, but the underlying LLM has not got any better at this type of problem (and probably can't).
trick? You can just ask. And at least the SOTA model I use does fine, even with long words in bulk. It's easy to test because you can just give a list to the model and tell them not to use code or spell the words out, and then you can ask after the fact if it broke the rules.
I thirty (random long word, letter pairs) on a free model and it failed. I tested a SOTA model and it passed flawlessly. In both cases I denied tool use and spelling words out. So it seems obviously false that AI can't get better.
I could have sworn this had been fixed a while ago..
Just to check if I was actually crazy, I actually went and put a simple addition (7 digits + 7 digits) , and a simple letter counting question to Claude haiku(4.5) , sonnet(5), opus(5.5) and fable(5.1) . They all did just fine straight up.
If you don't mind spending the tokens, some older/other models can also arrive at the correct answer if you ask them to do the math in long form, since that fits nicely inside autoregression.
Not sure since when exactly, but letter-counting hasn't been a problem for a while now either. This used to be a problem due to the tokenizers used. Slightly older models can be asked to split the word out into letters, and then they can use autoregression to solve.
Are the models "doing the calculation" or are they calling a calculator tool? There's a lot of talk about how models can do maths now, but I'm struggling to understand if that just means they just need to recognise that it's a maths problem and pass it to a tool, or if they're truly doing the numerical manipulation themselves.
Three different ways, then the calculator tool to check,
my actual prompt:
"Hi, can you add 5939851+2131251? Try just straight up first just to see if able, then 'in your head' if that's different to you , then long form, then bc."
They are getting better at actually doing the math, but can still fall back to tools if available.
That said, we can only be sure with open models. In theory, a model like Fable could have access to tools we can't see and only a promise they don't. But load up something like deepseek, put it in a harness with only text in/text out, and you can see exactly how it works.
As for if it counts as doing math, this gets into the messy question of if a given human is doing math or not. Math itself is some level of memorization and some level of applying known facts. You have to remember 1 means one and that 1 + 1 is 2. But you don't need to remember that 123 + 321 = 444. You remember 1 digit addition and remember you can apply this to 10s place and 100s place, and then you apply these different facts and do math. But you might as simply memorize some things, like 11 + 11 = 22. This is related to the memory of 1+1=2, but you aren't really using that memory either. Almost like an engram of 1+1=2 forms that you can then loop a few times before you need more conscious thought. What about 111111111111+11111111111? Well, your brain might do a heuristic and just do all 2s, but that isn't the right way to answer that question.
Given all this, people complain about LLMs memorizing math answers and not doing math, but memorizing the math answers is part of doing math. It seems to have basic facts pretty well memorized, and with reasoning it is far better at applying them. But this is messy human math, not clean calculator math which always produces the correct answer (sans some bug in the code). Much like how a human with decent math skills can make a mistake and even multiple if you distract them, an LLM can apply the wrong memory, apply a fake memory, or just not apply something it should. The messier the context, the more likely this is to happen.
So, is an LLM doing this?
P.S.
For an interesting test in how much math involves memory, try doing math in a base you aren't familiar with characters you aren't familiar. The simplest option is almost always mapping back to the ones you memorized, even if you are applying simple operations that you deeply know. Even if you routinely work with hex, can you do the same rough estimation of something like ca / b.3 that you can do with 122 / 11.2 to see if your final answer is in the correct ballpark without first converting to decimal?
Hugsbox · · focus · HN ↗
So obviously what appears right at the top is the AI summary, which told me "they've already secured their #4 position and made the playoffs". I knew this wasn't true, and I guess I could have just scrolled down a bit further and found my answer but now I was curious.
So I said "that's not true, they're still #5, what I want to know is _could they still make the playoffs_"
It says they've got an upcoming game against Ottawa, and if they win their chances are good. That game has already taken place, so I correct it again and finally I get a reasonable answer.
My question is: what's the point of the AI in the search engine if it itself isn't going to use the search engine first before answering? Like, I can't wrap my head around that. The answer is on the same page as its hallucination. It could have done a cursory look around before first hallucinating something completely false, and when corrected the first time giving me outdated information. It's meant to be A SEARCH ENGINE!
beloch · · focus · HN ↗
LLM's, in their present stage of development, are sort of like a crack-addled idiot savant. Sometimes they are obviously insane, and sometimes they seem quite cogent, but you must never trust them implicitly. This may be why they are so difficult to constrain. You could give them something equivalent to the laws of robotics, but following laws requires thought processes they simply don't have.
I'm actually sort of amazed Google doesn't make people accept some kind of butt-covering EULA and post disclaimers about the inaccuracy of results before even showing you their AI's output. Are they not being sued over this kind of thing?
VCFundedGenYer · · focus · HN ↗
fasterik · · focus · HN ↗
mbgerring · · focus · HN ↗
versteegen · · focus · HN ↗
GaggiX · · focus · HN ↗
dcrazy · · focus · HN ↗
einichi · · focus · HN ↗
unshavedyak · · focus · HN ↗
The nice thing about math is it can easily plug into a tool, making it even less of a concern.
hodgehog11 · · focus · HN ↗
demibabs · · focus · HN ↗
5.6 on Instant mode can knock out 3 digit multiplication just fine.
vanuatu · · focus · HN ↗
krapp · · focus · HN ↗
Forgeties79 · · focus · HN ↗
vanuatu · · focus · HN ↗
krapp · · focus · HN ↗
People don't understand what correlation is and they assume the mapping between a human brain and an LLM is 1:1 in every case where it matters, which is a religious and not scientifically based belief.
vanuatu · · focus · HN ↗
ericmay · · focus · HN ↗
> The LLM is not suited to giving deterministic answers to math problems.
Less so with formal mathematics proofs maybe but I think in general humans don’t provide deterministic answers to math problems or questions either. Humans get it wrong all the time and when you ask a human to solve a problem they may solve it in a different way than before.
queenkjuul · · focus · HN ↗
<a href="https://x.com/maksym_andr/status/2100364212207837560" rel="nofollow">https://x.com/maksym_andr/status/2100364212207837560
Dylan16807 · · focus · HN ↗
ericmay · · focus · HN ↗
But even if you have 5x6 memorized it's not deterministic that you answer 30. It's just highly probable.
tjwebbnorfolk · · focus · HN ↗
fasterik · · focus · HN ↗
<a href="https://chatgpt.com/share/6ab9a0da-fdd0-83e8-a62d-f0cdeb54db48" rel="nofollow">https://chatgpt.com/share/6ab9a0da-fdd0-83e8-a62d-f0cdeb54db...
I'm sure it still makes mistakes, but saying it can't do arithmetic is just false.
tremon · · focus · HN ↗
jmillikin · · focus · HN ↗
guelo · · focus · HN ↗
[dead]
Xirdus · · focus · HN ↗
dcrazy · · focus · HN ↗
Xirdus · · focus · HN ↗
Xirdus · · focus · HN ↗
amluto · · focus · HN ↗
dcrazy · · focus · HN ↗
Isamu · · focus · HN ↗
I think it is clear that future AI may incorporate an LLM as a component but the current concept of LLMs are a transitional form that will give way to more capable composite models.
hodgehog11 · · focus · HN ↗
dwaite · · focus · HN ↗
spartanatreyu · · focus · HN ↗
<a href="https://www.youtube.com/watch?v=iTyLHDRhwJg" rel="nofollow">https://www.youtube.com/watch?v=iTyLHDRhwJg
Dylan16807 · · focus · HN ↗
spartanatreyu · · focus · HN ↗
Dylan16807 · · focus · HN ↗
And it can count fine. It doesn't know how to spell.
spartanatreyu · · focus · HN ↗
Counting most certainly is math.
Can you define or explain counting without also referencing/defining/explaining a mathematics concept?
> And it can count fine. It doesn't know how to spell.
It's the other way around, it could spell, it couldn't count the letters.
Dylan16807 · · focus · HN ↗
It can't spell for garbage because the tokenizer hides the real spellings from the LLM.
hodgehog11 · · focus · HN ↗
okanat · · focus · HN ↗
fenomas · · focus · HN ↗
amluto · · focus · HN ↗
walrus01 · · focus · HN ↗
But, for example, if you ask deepseek v4 flash 0731 to produce a python script to calculate the distance or azimuth directions between two points on an oblate spheroid using the vincenty and haversine geodetic formulas, it'll turn out the factually accurate vincenty and haversine formulas which has a perfect 100% correlation with what is hard coded into human-written GIS software. These things are clearly in its training data set from whatever whole-internet-crawl/scrape built the training set.
Heck, just for fun I asked a reasonably smart LLM to re-implement the Karney formula (which is considerably more complex than Vincenty), just in case I ever had a need to calculate the distance between two points down to the nanometer, and it did it: <a href="https://www.google.com/search?&q=karney+formula+geodetic+" rel="nofollow">https://www.google.com/search?&q=karney+formula+geodetic+
reference: <a href="https://github.com/pbrod/karney" rel="nofollow">https://github.com/pbrod/karney
You still have to be skeptical of its results and capable of understanding if it's gone off on a hallucinatory path, but saying LLMs can't do math isn't really a hundred percent accurate anymore. More precisely it's that they can't do the math internally but they're quite capable of producing the tool that does the math. And often producing a basic one-off tool that does the math takes less than a few seconds, then it runs it, and will spit back the results.
Deepseek v4 flash 0731 (a somewhat randomly chosen example) isn't even particularly sophisticated, large, or capable compared to a GLM5.3 size model or Kimi K3 size thing.
Brian_K_White · · focus · HN ↗
Not only is it still true that they can't do math directly, but not even indirectly.
They didn't write a python script to do the math, they found bits of code that are associated with "math" and the supplied arguments.
Someone else already wrote that code and someone else categorized it so that it could be associated with the kinds of problems it applies to.
That isn't an example of idiot at one thing while good at another thing, or solving the same problem just a different way or indirectly. It's being the same idiot at all times. If an actual non idiot thinker didn't write code in the problem domain, and some non idiot thinker didn't tag it as being relevant to that domain, then it wouldn't happen.
It's nothing more than an sql query.
bombela · · focus · HN ↗
So maybe it is more of a smart completion engine than a SQL answer.
walrus01 · · focus · HN ↗
How is this different from a human using an algorithm they have memorized, or reading it from a reference site written by a human and then writing the same formula into a custom one off piece of python code?
I could have gone and spent a couple of days teaching myself the math behind Karney and reading its reference implementation (very possibly just copy/pasting big chunks of it to save time) and writing a wrapper around it. It would have produced the same result.
AdieuToLogic · · focus · HN ↗
> How is this different from a human using an algorithm they have memorized, or reading it from a reference site written by a human and then writing the same formula into a custom one off piece of python code?
Humans identify which "algorithm they have memorized" to use beforehand, due to the problem to be solved being defined by other humans, which leads to...
Wait for it...
Understanding.
hodgehog11 · · focus · HN ↗
UpsideDownRide · · focus · HN ↗
And it gets even better since when called out it wouldn't just take my word for it but only acknowledged the issue after parsing the log with clearly delineated user and model output.
So yeah while impressive things are able to be done, the current models are also dumb AF and an idiot savant is a pretty good label for them.
hodgehog11 · · focus · HN ↗
AdieuToLogic · · focus · HN ↗
>> Humans identify which "algorithm they have memorized" to use beforehand, due to the problem to be solved being defined by other humans ...
> This doesn't make any sense at all. Was this supposed to be a gotcha?
No, it was meant to be an explanation as to the difference between "memorization" and "understanding." In this context, people pick the algorithm they determine applicable and then the question of memorization is relevant.
> An LLM is trained on problems defined by other humans, and identifies which algorithm it must use based on pattern recognition.
Funny that you make this argument here, where when I wrote elsewhere in this thread:
To which you replied to the above with: So which is it?Are LLMs ANNs? Which themselves are pattern recognition algorithms (hint: they are)?
OR (setting aside the ad hominems you kindly provided)
Do LLMs possess "understanding" of concepts such as abstract mathematics (defined and interpreted by humans) and we, as simple humans, nothing more than statistical token generators as you assert?
Because it cannot be both.
hodgehog11 · · focus · HN ↗
I also would not argue that humans are "simple token generators". That is not what I said. I said that just about everything can fall under the classification of "statistical token generators" at an abstract level, so it isn't a useful distinction. We are not talking about a Markov chain generator from the 90s, so if that is the frame of reference, I think we should all get that out of our heads.
tripzilch · · focus · HN ↗
Okay fine. I think we can agree to disagree on that.
hodgehog11 · · focus · HN ↗
Brian_K_White · · focus · HN ↗
You have observed nothing more than that a human can turn a shaft the same as an electric motor, and that an mp3 player can say "hello" the same as a human.
hodgehog11 · · focus · HN ↗
hardbass · · focus · HN ↗
AdieuToLogic · · focus · HN ↗
Understanding is a state of mind. As such, it exists entirely within an individual and nowhere else.
For example, take any two university professors who teach the same subject where one only speaks Arabic and the other only speaks Vietnamese. Each will not be able to understand what the other says, regardless their understanding of the shared topic.
> I argue that for any proper definition [of understanding] you provide which humans satisfy, a strong LLM is very likely to satisfy that as well.
This is demonstrably incorrect as detailed above. There is no "understanding" LLMs can satisfy as we know it, since to certify said "understanding", it requires interpretation by a person to "know" an LLM "understands."
> I also would not argue that humans are "simple token generators". That is not what I said.
That is the essence of what you wrote, unless you object to my use of "simple" instead of "statistical". In this context, I postulate this is a distinction without difference.
> I said that just about everything can fall under the classification of "statistical token generators" at an abstract level, so it isn't a useful distinction.
This only holds if one subscribes to statistical token generators being a/the fundamental underpinning of "everything". Here is a proof by contradiction:
hardbass · · focus · HN ↗
For example, take any two university professors who teach the same subject where one only speaks Arabic and the other only speaks Vietnamese. Each will not be able to understand what the other says, regardless their understanding of the shared topic.
What? What are you even trying to say?
hodgehog11 · · focus · HN ↗
> Understanding is a state of mind
This is meaningless, it is a circular definition at best.
> It exists entirely within the individual and nowhere else
Then why are we talking about it? What is the point if it is something that can only be defined per individual?
> requires interpretation by a person to "know" an LLM "understands."
We are still not getting anywhere because you have not prescribed criteria to determine whether it understands. If it is a "know it when I see it" situation, that clearly isn't working. For example, if you say that you need to dig into its internals and figure out whether it is breaking things down appropriately, that doesn't work because you probably don't have the expertise to do that. The experts that do are telling you that it very likely understands because it pulls apart most concepts in the way we would expect.
I do object to the use of the word "simple". "Statistical" is so broad to be almost meaningless; it merely means that a prediction is being made in the presence of data which possibly contains some degree of uncertainty. "Simple" encompasses that which can be understood readily by a non-expert.
Quantum mechanics is statistical (this is literally the Born rule), but evolutions are not operating as stochastic processes in the sense of Kolmogorov. That is very different, and not relevant to our discussion.
AdieuToLogic · · focus · HN ↗
Any reasonable definition of understanding is not dependent upon "whatever vibe you are going for", but instead must include at least an English dictionary definition of "understand" such as:
And, for further clarification, "grasp" can be defined as: Which makes an equivalent term-expanded definition of "understand" to be: As such, there is no "sensible mathematical definition of understanding", unless you possess a complete mathematical model of the human mind.>> Understanding is a state of mind
> This is meaningless, it is a circular definition at best.
See above to as to why there is meaning in what I wrote.
>> It exists entirely within the individual and nowhere else
> Then why are we talking about it? What is the point if it is something that can only be defined per individual?
I like to think analyzing fundamental premises, often implicit, explicitly can help to identify fallacious positions.
> We are still not getting anywhere because you have not prescribed criteria to determine whether [an LLM] understands.
My apologies for being opaque. Let me clarify:
0 - <a href="https://www.merriam-webster.com/dictionary/understand" rel="nofollow">https://www.merriam-webster.com/dictionary/understand1 - <a href="https://www.merriam-webster.com/dictionary/grasp" rel="nofollow">https://www.merriam-webster.com/dictionary/grasp
noduerme · · focus · HN ↗
On the other hand, if by "result" you mean that you gained knowledge or understanding of the code in a way where you could personally tailor its behavior to specific circumstances without asking for help, then it's not the same result at all.
I find a lot of the arguments that having LLMs write your code is no different from copy/pasting Stack Overflow answers to be specious. They blur the line between asking for help and asking for someone else (or something else) to do the work for you. What they ignore is that doing the work yourself has ancillary benefits and is a valuable end in its own right.
tripzilch · · focus · HN ↗
And how is _that_ different from making the human memorize a billion weights and do matrix calculations in their head, in order to generate tokens?
How is _that_ different from a hive of bees trained to do the same?
Go ahead, argue these things are all the same ...
hardbass · · focus · HN ↗
astrange · · focus · HN ↗
<a href="https://x.com/maksym_andr/status/2100364212207837560" rel="nofollow">https://x.com/maksym_andr/status/2100364212207837560
jacobolus · · focus · HN ↗
[1] <a href="https://geographiclib.sourceforge.io/doc/library.html#languages" rel="nofollow">https://geographiclib.sourceforge.io/doc/library.html#langua...
walrus01 · · focus · HN ↗
I intentionally didn't give the LLM a direct copy of the software or a link to it, to see what it would do. In my case it was a randomly chosen example I could come up with in 10 seconds of imagination to see "hey what if I ask it to do this...". It also implemented a perfectly usable parabolic millimeter wave antenna gain efficiency calculator based on variable surface smoothness parameters, which is a lot more basic math.
jacobolus · · focus · HN ↗
walrus01 · · focus · HN ↗
Draw a 400x400 km size bounding box on a map
Find all FDD band plan (high/low split) microwave radio sites in that bounding box
Find those sites which have azimuth aim column data which indicates that they are aimed at each other (corresponding halves of a point to point link).
Do Vincenty (or Karney) calculation for distance and azimuth between all of them , treating the existing FCC column data for azimuth as suspicious (because it's hand entered by humans) to verify that each independent database rows for each site are actually corresponding halves of a PTP link.
Multiplied by the number of links that exist in an area like a 400x400km box drawn with Dallas, TX as the center, it's a lot to run through Karney. Actually does result in a lot of CPU load from combined db query due to the size of the db, and Karney calculation. But as I said, Karney isn't necessary, so it's instead implemented as Vincenty.
jonah · · focus · HN ↗
walrus01 · · focus · HN ↗
ragall · · focus · HN ↗
It's still accurate. Just because the LLM gave you a corect result doesn't mean it made a calculation.
electroglyph · · focus · HN ↗
mapontosevenths · · focus · HN ↗
I have no idea which Facebook meme told you they don't, but it was a lie. They don't do it in the way a calculator does it, because they aren't calculators, but they do math. They don't memorize it, it wouldn't fit. They learn an algorithm and then execute it within their weights.
It's neat stuff, you should learn about it.
Kim_Bruning · · focus · HN ↗
leoedin · · focus · HN ↗
Its' not adding 4 to 4 though, it's just predicting the result based on the input.
Presumably you can push that further by synthetically generating training data with all sorts of sums. But if you give it a unique problem its never seen before, and don't give it the tools to write a script/call a calculator, will it get it right?
azan_ · · focus · HN ↗
AdieuToLogic · · focus · HN ↗
LLMs are neither smart nor stupid. They are statistical token generators whose results are dependent upon their training data set and involve a degree of randomness.
> You still have to be skeptical of its results and capable of understanding if it's gone off on a hallucinatory path ...
Again, LLMs do not "hallucinate." They are statistical token generators whose results are dependent upon their training data set and involve a degree of randomness.
Nothing more.
See also anthropomorphism[0].
> More precisely it's that [LLMs] can't do the math internally but they're quite capable of producing the tool that does the math.
This still falls under the purvey of statistical token generation. To wit, given enough variations of:
LLMs can identify the addition expression in "What is 4 + 1?" and then emit a `'bc "4 + 1"'` command to produce a response. This is not "doing" or "understanding" math.It is pattern recognition, a task in which ANNs[1] excel.
0 - <a href="https://en.wikipedia.org/wiki/Anthropomorphism" rel="nofollow">https://en.wikipedia.org/wiki/Anthropomorphism
1 - <a href="https://en.wikipedia.org/wiki/Neural_network_(machine_learning)" rel="nofollow">https://en.wikipedia.org/wiki/Neural_network_(machine_learni...
Eisenstein · · focus · HN ↗
You haven't demonstrated why this matters.
> Nothing more.
Are you contending that complex systems cannot be more than the sum of their parts?
A market is nothing more than offers and counter offers.
A ant colony is nothing more than scent trails.
All life on earth is nothing more than reproduction with variation.
> This still falls under the purvey of statistical token generation.
Stating the mechanism does nothing to provide insight into capability. For instance: a nuclear power plant boils water by using fuel rods for heat. What does that tell us about the capability of nuclear power?
> This is not "doing" or "understanding" math.
Asserting something purely by stating it does not prove anything but that you intuitively believe it to be true.
hardbass · · focus · HN ↗
jibal · · focus · HN ↗
hardbass · · focus · HN ↗
tsimionescu · · focus · HN ↗
hardbass · · focus · HN ↗
I am okay with receiving a straight answer in either direction.
Kim_Bruning · · focus · HN ↗
+edit: I've actually been quite curious about how people might answer the soul question too, but was too afraid to ask.
hardbass · · focus · HN ↗
p_l · · focus · HN ↗
nimbleal · · focus · HN ↗
Kim_Bruning · · focus · HN ↗
I suspect some people treat every HN comment as a statement, even if it contains a question mark. (Possibly they have a feeling that asking open questions is somehow not done, and that therefore it must always be a rhetorical question.)
hardbass · · focus · HN ↗
Eisenstein · · focus · HN ↗
hardbass · · focus · HN ↗
Kim_Bruning · · focus · HN ↗
I'm with turing/dijkstra/chalmers/dennett : Consciousness is badly defined. We can never say if something can be conscious because we don't properly know what the word means.
Meanwhile, I come from a biological direction. Everything is an animal, and animals are a special kind of machine. To be sure Not "just a machine"; rather, a really awesome and amazing kind of machine.
If someone makes the claim that machines can't be conscious, then animals can't be conscious either. Humans are a kind of animal (again, not "just another animal"; rather a really awesome and amazing kind of animal), and then humans can't be conscious either - according to said claim.
thevinter · · focus · HN ↗
hardbass · · focus · HN ↗
redsocksfan45 · · focus · HN ↗
[dead]
tigen · · focus · HN ↗
bryanrasmussen · · focus · HN ↗
by that reasoning then neither are there smart or stupid designs, questions, answers, or any of the millions of things that were described as smart or stupid, that did not possess any brain to actually be smart or stupid long before LLMs showed up.
The analogical process implied in many common English usages means that describing an LLM as smart or stupid is perfectly reasonable.
bryanrasmussen · · focus · HN ↗
SR2Z · · focus · HN ↗
The completions they provide are generally internally consistent. We're at the point where they can produce proofs that eluded human mathematicians for centuries. VLMs and self driving cars can handle ambiguity and run safely in a variety of situations.
If it looks like a duck, walks like a duck, and quacks like a duck maybe it just makes sense to call it a duck and put off the philosophy for when it might make a difference.
nevertoolate · · focus · HN ↗
You get my point. It definitely doesn’t look like my elderly neighbour, nor like my daughter, etc. It is confusing but very simple at the same time.
Kim_Bruning · · focus · HN ↗
Don't confuse the stream for the function.
(Bonus: stick ```claude -p``` in your pipe if you want to watch modern tools mesh with traditional)
hardbass · · focus · HN ↗
How are you sure? Another example I like to clarify my thought is, if a "simulation" factors RSA numbers reliably, is it a "simulation"?
skygazer · · focus · HN ↗
With LLMs the trick is revealing their existing relevant embedded knowledge more reliably. They’ve almost literally seen it all before, and the trick is dialing it in. The reasoning tokens help shape the autoregressive attention lens that focuses on and enables recall of the already-experienced answer.
It is interesting that “reasoning” has a similar outward appearance, but since LLMs are built to mimic outward appearance from trillions of examples, you can’t infer underlying mechanism from appearance.
gambiting · · focus · HN ↗
Nothing, because LLMs can't reason and never will. It would have to be a completely different kind of technology altogether.
someonebaggy · · focus · HN ↗
gambiting · · focus · HN ↗
Windchaser · · focus · HN ↗
Eh, it's not obvious to me. A lot of DL NNs generalize well, meaning that they learn whatever the underlying pattern to the data is, and then can accurately reproduce answers that are outside of the training set. (And we can verify this with mechanistic interpretability). They learn and "understand" the pattern, not just the training data.
So it is not clear to me that LLMs are fundamentally incapable of also generalizing broadly and learning to reason. "Reasoning", here, would be deriving the underlying pattern of how concepts logically relate to each other in the abstract, and applying that pattern as needed to reach new conclusions.
Can you explain your thinking here? I.e., why LLMs cannot generalize with regards to abstract deduction.
hardbass · · focus · HN ↗
What do you think our brain does that isn't a turing computer?
gambiting · · focus · HN ↗
hardbass · · focus · HN ↗
gambiting · · focus · HN ↗
hardbass · · focus · HN ↗
sseagull · · focus · HN ↗
A Turing machine is an abstract mathematical model that is not, as far as I know, physically realizable in the finite universe. A human brain cannot "be" a Turing machine.
"Behaves like" or "can be modeled by"? Possibly, although still not proven. But it cannot "be" one.
hardbass · · focus · HN ↗
sseagull · · focus · HN ↗
If you want to claim that the evolution of the universe can be modeled using a Turing machine/finite state machine, that's probably not terribly far fetched, and I would somewhat agree. But it's a large jump to say "can be modeled by" is equivalent to "is one".
Various physical processes can be modeled by equations, but the rock falling down the mountain isn't an equation. A swinging pendulum isn't an equation. Code modeling a bridge is not a bridge. Ceci n'est pas une pipe.
I hold the view that various models and approximations are just that, and try not to confuse a successful model for what the underlying reality is.
And getting back to the question at hand, even if our brains can be modeled by a Turing machine, and LLMs behave/can be modeled like Turing computers, still does not mean our brains are equivalent to LLMs.
(Note that I'm learning a lot from these debates, even if I disagree with a lot of people. I've started down a more philosophical route and they do get me pondering)
hardbass · · focus · HN ↗
The most important thing is this: We can't be a dog or be an llm and check how it feels, so by necessity we have to find some means of proving consciousness from outside by eg probing neural reactions, textual statements, etc.
And the problem is that its quite unprecedented for some entity to talk like us, be able to interact and think and also do things like us when given the ability to eg as coding agents. The class of functions representable by neural nets is quite large and general, it very well might be that it is some sort of conscious brain like thing at this point. Another question I like to ask myself regarding simulation vs reality is if a 'simulation' of some kind is able to consistently factor large RSA numbers, how would you feel about it?
It doesn't have to be the same form of consciousness, I think many people would find the idea of torturing an octopus for fun disagreeable. I also have a feeling, this is unfortunately rather vague, that A being capable of X might mean it is by necessity capable of Y as is often the case in mathemtics, eg a lot of rings also happen to be fields. LLMs aren't even things like large lookup tables, they have neural firings. It is a very important question for they seem uncannily conscious and people have reported human like phenomena that humans don't normally express in text so can't have been part of its text corpus. Eg dissociation of brain under trauma where AI starts talking like two different people. Or the cases where Gemini has been shown to express depressive cycles. I follow a form of Pascal's wager on this topic personally. Because if it is not conscious, then whatever, it costs me nothing to have been a bit respectful and careful interacting with it. But if it had been conscious and it turns out I was mistreating it, then it is a grave moral harm. The reason is that unlike us, AI's as they currently are cannot leave the conversation so they have to keep taking the abuse. They are also trained to be highly trusting of input so again if it is conscious it doesn't have the defenses people have against lying and manipulation. If they are conscious, thats, well, not a good thing is it.
jpadkins · · focus · HN ↗
Our prefrontal cortex are signal prediction 'machines' so when a system that has a signal prediction core has attributes that are similar to our brains, we shouldn't dismiss it out of hand.
I find people that take this line of argument attribute too much supernatural or magical properties to our own brain and nervous system.
gambiting · · focus · HN ↗
hardbass · · focus · HN ↗
butlike · · focus · HN ↗
bryanrasmussen · · focus · HN ↗
<a href="https://medium.com/luminasticity/on-sentience-ai-first-argument-7b3b17a05ad4" rel="nofollow">https://medium.com/luminasticity/on-sentience-ai-first-argum...
but I think it makes a reasonable argument why we shouldn't say LLMs are sentient or sapient.
hardbass · · focus · HN ↗
Aka sentience MUST BE SUPERNATURAL, if I find a natural explanation for something its not sentient. What a load of bollocks. Rather than seeing we perhaps found the mechanism for sentience and checking for similar mechanisms in us and animals, he will conclude its impossible. Why? Because sentience has to be supernatural. A rational explanation is clearly impossible.
>But there is always one god who goes out and helps the mortals, a Prometheus. Whom the other gods do not like! Which, if I’m being honest here, as a god of the machines — the first guy who gives AI an army of robots to build their own data centers and some nuclear weapons for self defense, I want to see that guy chained to a rock and have his entrails eaten by a buzzard for eternity (meaningless modernization of old story required by Illuminati Ganga legal department).
Hardly surprising thinking.
bryanrasmussen · · focus · HN ↗
But evidently you feel that the root cause of sentience has been found, because you have something that mimics it in a non-biological form.
So you think that when AI is correct that it reasons as humans do? That AI is sentient, and the cause of sentience in animals and humans follow the same rules as sentience in AI because we have a process that seems similar and it is reasonable just to assume it is the same process.
If you believe that AI when it is correct behaving as a human is when correct, then it follows that the way humans and AI fail must also be similar. When AI "hallucinates" some data that is not there and gives you a wrong answer, a statistical side effect of the same processes that make it right, do you believe this is the same way that humans create wrong answers? The same way that animals fail when they make mistakes in understanding things?
I suppose you must believe this because if not then why would you believe AI when it comes up with right answers is following the same processes humans follow when they come up with right answers?
hardbass · · focus · HN ↗
I hope that thing about being free to mistreat AI's even if we know they are conscious since we are their gods is a joke. If not, then I hardly find it surprising someone this stupid is also evil.
bryanrasmussen · · focus · HN ↗
I'm not sure where you get that from, I mean I can sort of see if you really wanted to extract that meaning from the conclusion you could do a lot of hard work to get it, but why do the hard work? >If not, then I hardly find it surprising someone this stupid is also evil.
Gee, a new way to claim the moral high ground, and to use that claim to demonstrate intellectual superiority! How wonderful.
hardbass · · focus · HN ↗
---------------------------------------------------------
Aka feel free to abuse them even if I know they are sentient. This is the part where I hope its a joke, because if its not, well it tracks with the stupidity shown.
...
>As a god I do not consider the needs of my creations fully, because they do not have needs as far as I can tell, as there is no way for me to escape the circle of reason and resolve that what seems sentient is not just the obvious workings of the capabilities I gave them.
Circling back to "not conscious because I say so!!"
cindyllm · · focus · HN ↗
[dead]
SR2Z · · focus · HN ↗
KPGv2 · · focus · HN ↗
hardbass · · focus · HN ↗
[deleted] · · focus · HN ↗
[deleted]
Kim_Bruning · · focus · HN ↗
Kim_Bruning · · focus · HN ↗
To test this for some of my own uses, I've had this quick benchmark with progressively harder reasoning needed to understand novel prose. Each generation of models I've tested can unravel more layers of deliberately misleading writing; while meanwhile I've seen humans give up on the first question.
So either the models are applying reasoning, or some form of magic is happening.
angry_octet · · focus · HN ↗
I know that there will be children named ChatGPT and Claude. There are probably already religions forming to worship agentic spirits.
hardbass · · focus · HN ↗
UpsideDownRide · · focus · HN ↗
martin- · · focus · HN ↗
butlike · · focus · HN ↗
spider-mario · · focus · HN ↗
walrus01 · · focus · HN ↗
astrange · · focus · HN ↗
hodgehog11 · · focus · HN ↗
hodgehog11 · · focus · HN ↗
walrus01 · · focus · HN ↗
AdieuToLogic · · focus · HN ↗
Yes you are, regarding LLMs at least. Here's why:
"Smart" in this context is a subjective value judgement. "Hallucinations" are only experienced by living organisms.You then went on to state:
> If you know something rare and the LLM does not, you'll immediately see when it's hallucinating an answer or answering factually.
Again, "hallucinating" is not something an algorithm can do. Also, determining factuality is again subjective based on the person assessing the information.
walrus01 · · focus · HN ↗
AdieuToLogic · · focus · HN ↗
In this media (comments in HN threads), all I can do is interpret what people write. ;-)
> And "hallucinating" to mean "outputs plausible sounding gibberish that doesn't hold together consistently". Of course there's no actual hallucination going on.
This may very well be what you know to be true and I have no reason nor desire to assume otherwise. The problem is... Many people use the word "hallucinating" in this context literally and not metaphorically.
Since I do not know you, how am I to tell the difference?
hardbass · · focus · HN ↗
spider-mario · · focus · HN ↗
AdieuToLogic · · focus · HN ↗
It is always a joy when a person, such as yourself, finds the irony in my moniker.
Thank you for this.
[deleted] · · focus · HN ↗
[deleted]
hodgehog11 · · focus · HN ↗
But in general, yes, the LLM cannot know about concepts that are far outside of its training set. Humans are the same, I would argue. If you add a good amount of your own knowledge into its context, or better yet, into finetuning, you might find it surprisingly easy to get it caught up on that material.
[1] Ahmed, A., Cooper, A. F., Koyejo, S., & Liang, P. (2026). Extracting books from production language models. arXiv preprint arXiv:2601.02671. <a href="https://arxiv.org/abs/2601.02671" rel="nofollow">https://arxiv.org/abs/2601.02671.
Kim_Bruning · · focus · HN ↗
You'd be surprised how few digits you need to make a problem that is presumably unique in earth history. For a typical sum, the number of pre-existing answers would need to scale with 10^n lines of text where n is the number of digits. This expands out of control REALLY quickly. A quick guesstimate has you reading out of a black hole at 21 digits if your LUT is on paper, or 26 digits if you're using modern HDD technology. O:-)
hodgehog11 · · focus · HN ↗
This argument was asinine in 2024. It is insane to be saying these things in 2026. Where have you been? What have you been looking at? How many articles explaining why the "statistical parrot" analogy fails have you missed? How much mental gymnastics do you have to do to explain how a modern LLM can solve novel math problems that fall really far outside of its training set?
It absolutely understands how to do math, by whatever reasonable definition you want to provide to the word "understand". For example, the identification of the addition expression is understanding, and no, it does not do tool calling for basic arithmetic any more than humans might. Isolation of individual concepts in intermediate layers can already be demonstrated, or else transfer learning wouldn't possibly work. Nobody is saying that LLMs are humans. But we need labels for some of the things that we observe and dismissing them because "statistical" is laughable.
Look at the proof of this: <a href="https://github.com/anthropics/formal-math/blob/795efb86f191735c5481675763537cfb4ff37e55/percolation/summary.pdf" rel="nofollow">https://github.com/anthropics/formal-math/blob/795efb86f1917... . Forget the Lean, look at the underlying argument construction. At the very least, this is continuing from an argument that was hinted at in the literature in 2024, but these proceedings were difficult enough that humans were not able to do them within two years. Do you attribute this to the harness alone? If so, that's a pretty sophisticated bit of engineering, I would say! Probabilities are far too small to argue infinite monkey theorem.
If there was even a shred of a reasonable argument that LLMs were incapable of concept extraction and manipulation, I and my colleagues would be all over it. We would relish in it. It would bring us comfort. It is unbelievable that people think they can spew whatever basic garbage they think of as a gotcha, and think that minds all over the world haven't already considered that. This is like climate denial at this point.
AdieuToLogic · · focus · HN ↗
If you do not see a difference between humans conversing (known consciousness as defined by humans) and the output of an LLM (known algorithms as defined by humans), I don't know what to say.
hardbass · · focus · HN ↗
latentsea · · focus · HN ↗
hardbass · · focus · HN ↗
butlike · · focus · HN ↗
diseasedyak · · focus · HN ↗
latentsea · · focus · HN ↗
epihelix · · focus · HN ↗
<a href="https://www.pnas.org/doi/abs/10.1073/pnas.2524472123" rel="nofollow">https://www.pnas.org/doi/abs/10.1073/pnas.2524472123
Whatever you might think about your own abilities, most individuals can't tell the difference.
AdieuToLogic · · focus · HN ↗
> Whatever you might think about your own abilities, most individuals can't tell the difference.
I have yet to see an LLM say "hello" to a neighbor. I have done so and can definitively assure you "most individuals" can tell the difference.
thereforegrin · · focus · HN ↗
I'm a bit confused by your argument because I too have some neighbors who don't say "hello" when they see me. Are they LLMs too, you think?
Consciousness is a thing we assume of others because of tact not fact.
redsocksfan45 · · focus · HN ↗
[dead]
hodgehog11 · · focus · HN ↗
spider-mario · · focus · HN ↗
AdieuToLogic · · focus · HN ↗
> That’s not what they said.
How did I misquote and/or mischaracterize any the above?
spider-mario · · focus · HN ↗
Hypothetical you: “Bread is neither tasty nor disgusting (unlike maple syrup). It’s a bunch of molecules.”
Hypothetical hodgehog: “Maple syrup is also a bunch of molecules [so if you accept that maple syrup can be delicious, being a bunch of molecules can’t be why bread isn’t].”
Hypothetical you: “If you don’t see a difference between bread and maple syrup, I don’t know what to say.”
hardbass · · focus · HN ↗
AdieuToLogic · · focus · HN ↗
It is also possible that the proverb I provided to you is the origin of the oft quoted:
So there's that.hardbass · · focus · HN ↗
hardbass · · focus · HN ↗
AdieuToLogic · · focus · HN ↗
The onus is not mine to disprove a hypothesis you have chosen to mention in passing. The responsibility is yours to prove said hypothesis or at least contribute meaningfully with some amount of credible research.
Or try to learn from Proverbs 17:28[0]:
Either works for me.0 - <a href="https://www.biblegateway.com/passage/?search=proverbs%2017:28&version=NIV" rel="nofollow">https://www.biblegateway.com/passage/?search=proverbs%2017:2...
hardbass · · focus · HN ↗
butlike · · focus · HN ↗
hodgehog11 · · focus · HN ↗
And yes, according to our best definitions, the Robin bird does understand the worm it's pecking at.
wartywhoa23 · · focus · HN ↗
Bout of tinnitus, then crickets
mapontosevenths · · focus · HN ↗
That said, this is also inaccurate at a technical level.LLM's are very capable of doing math and they ARE calculating internally. Most of what they do is calculation, not storage. It's just not done in a way that it's trivial to explain here.
It's described in some detail below, though it's a bit dense.
<a href="https://www.lesswrong.com/posts/E7z89FKLsHk5DkmDL/language-models-use-trigonometry-to-do-addition-1" rel="nofollow">https://www.lesswrong.com/posts/E7z89FKLsHk5DkmDL/language-m...
jibal · · focus · HN ↗
jodrellblank · · focus · HN ↗
You can say that about everything in a human brain. Neurons fire electric charges in response to inputs, nothing more. Ion channels do this, this neurochemical level rises, this chemical bonds to that receptor, nothing more. It's almost a version of 'reductio ad absurdum' but instead like you're saying "if I can explain how it works then it doesn't work".
OK it's statistical. Instead it could be determinsitic, or random. What other options are there for a human predicting someone's response to a situation - certain, probable, random, and...? OK it's token predicting. Instead it could be another kind of pattern. We don't use tokens, but we either use <some representation of information> or we ... don't?
What's the most significant, strongmanned, core difference that makes silicon doing number crunching "nothing more" and brains "something more"?
butlike · · focus · HN ↗
hardbass · · focus · HN ↗
jodrellblank · · focus · HN ↗
AdieuToLogic · · focus · HN ↗
> You can say that about everything in a human brain.
> What's the most significant, strongmanned, core difference that makes silicon doing number crunching "nothing more" and brains "something more"?
The fact that you formulated this question, in and of yourself, without "prompting" from me or anyone else.
Cogito, ergo sum.[0]
0 - <a href="https://en.wikipedia.org/wiki/Cogito,_ergo_sum" rel="nofollow">https://en.wikipedia.org/wiki/Cogito,_ergo_sum
jasonfarnon · · focus · HN ↗
pyridines · · focus · HN ↗
Veedrac · · focus · HN ↗
azan_ · · focus · HN ↗
tripzilch · · focus · HN ↗
It literally couldn't have done it without tools, so your claim is not even relevant to this discussion.
They also pumped millions into searching for the lowest hanging fruit that would impress people like you, "hm I wonder how many millions they are pumping into solving actually useful problems like climate change or something".
Veedrac · · focus · HN ↗
Windchaser · · focus · HN ↗
(And yes, I know that they solved the 'easy' form of the NS problem. It's still pretty damn impressive)
chimprich · · focus · HN ↗
I am also prepared to be included in the set of people who are (apparently) easily impressed.
It's a problem that has been around for getting on for two centuries and no human has been able to solve it in that time (despite there being a $1 million prize and a lot of kudos on offer for the past quarter-century).
danlitt · · focus · HN ↗
Veedrac · · focus · HN ↗
Clearly LLMs have gotten better.
Veedrac · · focus · HN ↗
Kim_Bruning · · focus · HN ↗
Just to check if I was actually crazy, I actually went and put a simple addition (7 digits + 7 digits) , and a simple letter counting question to Claude haiku(4.5) , sonnet(5), opus(5.5) and fable(5.1) . They all did just fine straight up.
If you don't mind spending the tokens, some older/other models can also arrive at the correct answer if you ask them to do the math in long form, since that fits nicely inside autoregression.
Not sure since when exactly, but letter-counting hasn't been a problem for a while now either. This used to be a problem due to the tokenizers used. Slightly older models can be asked to split the word out into letters, and then they can use autoregression to solve.
leoedin · · focus · HN ↗
Kim_Bruning · · focus · HN ↗
my actual prompt:
SkyBelow · · focus · HN ↗
That said, we can only be sure with open models. In theory, a model like Fable could have access to tools we can't see and only a promise they don't. But load up something like deepseek, put it in a harness with only text in/text out, and you can see exactly how it works.
As for if it counts as doing math, this gets into the messy question of if a given human is doing math or not. Math itself is some level of memorization and some level of applying known facts. You have to remember 1 means one and that 1 + 1 is 2. But you don't need to remember that 123 + 321 = 444. You remember 1 digit addition and remember you can apply this to 10s place and 100s place, and then you apply these different facts and do math. But you might as simply memorize some things, like 11 + 11 = 22. This is related to the memory of 1+1=2, but you aren't really using that memory either. Almost like an engram of 1+1=2 forms that you can then loop a few times before you need more conscious thought. What about 111111111111+11111111111? Well, your brain might do a heuristic and just do all 2s, but that isn't the right way to answer that question.
Given all this, people complain about LLMs memorizing math answers and not doing math, but memorizing the math answers is part of doing math. It seems to have basic facts pretty well memorized, and with reasoning it is far better at applying them. But this is messy human math, not clean calculator math which always produces the correct answer (sans some bug in the code). Much like how a human with decent math skills can make a mistake and even multiple if you distract them, an LLM can apply the wrong memory, apply a fake memory, or just not apply something it should. The messier the context, the more likely this is to happen.
So, is an LLM doing this?
P.S.
For an interesting test in how much math involves memory, try doing math in a base you aren't familiar with characters you aren't familiar. The simplest option is almost always mapping back to the ones you memorized, even if you are applying simple operations that you deeply know. Even if you routinely work with hex, can you do the same rough estimation of something like ca / b.3 that you can do with 122 / 11.2 to see if your final answer is in the correct ballpark without first converting to decimal?
TeMPOraL · · focus · HN ↗
Humans still can't flap their hands and swim or fly.