When thinking about ‘AI’ I sometimes find it useful to replace the terms ‘model’ or ‘AI’ or whatever with a description of what we’re really talking about here: a whole bunch of floating point numbers.
> In a series of private meetings, the company consulted religious scholars to help instill morality into its FLOATING POINT NUMBERS — and make the case that their numbers could be conscious.
I recommend keeping a few NaNs in-hand if you’re worried. Spread a couple of around if your numbers start to stir and you’ll be right as rain.
> It appeared to Rabbi Navon that Mr. Olah and his team believed that FLOATING POINT NUMBERS had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
They’re cleverly designed and work well for their purpose, but can assure you they have neither dignity nor moral status.
> If the FLOATING POINT NUMBERS themselves could be made to choose goodness, Mr. Olah reasoned, the world would be more safe.
I get it. I’ve spent years trying to get my floating point numbers to choose goodness - I wish them luck.
that’s like the whole philosophical discussion. where on the line is this llm; closer to a rock or a human? a plant is closer. and a fly, a lizard. we still treat so many things terribly.
It does seem exceptionally silly that a collection of floating point numbers has immediately outranked animals and other living beings. That’s why i like the parent comment. They’re numbers!
edit: maybe easier to say collections of atoms though are not to be treated equally. That’s reasonable isn’t it?
It is a philosophical discussion. It has also been debated by great minds and the consensus right now is that we do not know (and there is a possibility that it is unknowable) when something is conscious.
Any determination is fundamentally flawed if it is derived from assuming that we know the absolute truth or falseness of the proposition that an algorithm can think.
It doesn't matter what you believe or feel or think, there is no consensus. If you are prepared to accept the belief that an individual can unilaterally decide the person-ness of something, then be prepared to accept when some individual decides to act on the principle that you don't have it.
> If you are prepared to accept the belief that an individual can unilaterally decide the person-ness of something, then be prepared to accept when some individual decides to act on the principle that you don't have it.
No, I will not be prepared to accept that, and neither will I accept that. This sounds like a threat, and a really funny one at that. You must accept that my floating numbers might be persons, otherwise... somebody will say that you're not! And then something bad might happen, who's to say.......
In the civilized world we all agree on the personhood of humans. If I say that your chatbot isn't a person, I am not uprooting the entire social understanding of personhood - I am not saying that suddenly you might not be a person, or that guy might not be a person. I'm saying that I refuse to consider a chatbot a person.
It's as if I were to come up to you with a brick in my hand, saying that this brick is a person. If you were to say "no", I would reply "well, if this brick isn't a person, your chatbot isn't a person either!"
How did you make such a big leap from consciousness to personhood anyway? They are two quite different things. Consciousness is a question of science, personhood is a question of culture. Pigs might be conscious and we treat them in a disgusting way, but even if we achieved a worldwide vegan utopia, wouldn't we still treat them differently from humans?
I hate to use Claudish here, but "postnatal" and "dead" are quite load-bearing here :) Clinically dead? Coma? Born human baby, but with anencephaly - expected lifespan is days, and there's literally no brain - but, well, are they a person? Ask 1000 people - and I'm betting we'll get a nice bimodal distribution.
I take your point that a human walking around a city will - out of that 1000 responses - hopefully get 990+ of "yes, a person" - but that will start to degrade quickly once you hit boundaries.
And the less said about dehumanizaion - untermensh and that entire line of thinking - the better.
>In the civilized world we all agree on the personhood of humans.
You seem to have put a second tautology in to your tautology.
All right thinking people think rhat the things that qualify qualify.
If they didn't they wouldn't qualify, or be right thinking. But that's ok because we know the things that don't qualify don't qualify if we didn't know that we wouldn't be right thinking.
>How did you make such a big leap from consciousness to personhood anyway?
Once you start unilaterally making decisions about the unproven, you can decide all of the unprovables. History shows that personhood rolls into the firing range fairly quickly when people start doing that.
I'm more than comfortable saying you can't do that before someone openly declares that they are doing it.
I'm sorry but I don't understand your first part. Give it to me in plain English, doc.
>Once you start unilaterally making decisions about the unproven, you can decide all of the unprovables.
You mean that if I say your dumbass chatbot isn't conscious, it's like a slippery slope and next thing you know you're not a person anymore, and you're in a concentration camp? Really, man?
If things we understand can't instantiate consciousness then I reckon we better shut down science as quick as we can, who knows how long it'll be before we understand something important!
I'm pretty sure it's possible to make conscious machines and I think it's very likely we'll eventually figure out how. The point about the spec is not "it isn't conscious because we know how it works" but "we know it isn't conscious because of what we know about how it works".
More likely we will finally settle on something like “it isn’t conscious because only humans can be conscious”.
In the past, we’ve treated even other humans poorly because we think they’re different in meaningful ways, I sadly don’t see how this will be different with AI.
I'm not sure who the "we" in this is, but it's certainly not my position. I'm pretty sure most if not all animals are conscious. I don't think that means we shouldn't kill, eat and use them, but I do think it means industrialised animal farming as commonly practiced is a cosmic horror that is difficult to
even contemplate.
I think we have an extremely strong basis on which to say AI as it currently exists does not have conscious experiences, though, and if we start trying to pretend they do some real absurdities will result. Indeed the consequences could be pretty terrible in terms of power concentration, and I doubt it's a coincidence that those best placed to form a new priesthood of AI and tell
everyone how it should and should not be treated are pushing the idea so strongly.
Does it even matter? Turns out you don’t need to be conscious to do useful intelligence based work.
Same for all the people who argue that they aren’t “intelligent” and don’t “know” things. Either they are/do, or that stuff isn’t as important as you’d think.
The only way it would matter is if you wanted to moralize about enslaving your computer or what it means to restart it.
> if you wanted to moralize about enslaving your computer
That is precisely the context of the debate. You seem to find it absurd, as do I, but the reason we're talking about whether LLMs are conscious in this thread has nothing to do with whether they are useful. It's because important people, such as Dario Amodei, are engaging in exactly the kind of moralising over "AI welfare" that you're scoffing at.
Dario and friends are just marketing. Getting the pope to say your chatbot is a real boy would be the ultimate flex. And it’s clearly working judging by all the attention it gets them.
Considering a human body consists of 10^28 atoms, it is 15 orders of magnitude different. The emerging properties of 10^23 parameters may be much better defined without needing to know the properties of each parameter.The theology of such a big llm would be interesting. Current ones, not so much.
That’s overly reductive and thus completely misses the problem entirely.
What are biological neurones if not electric impulses of varying strengths?
We need to stop thinking of LLMs as purely a piece of software and ask those hard-to-define philosophical questions if we want any hope of building safety into our models.
Just because consciousness is something we don't fully understand doesn't mean it's the same thing as something else that's similar that we understand. That's a cargo-cult. The stone age Melanesian islanders making air strips out of bundles of weeds and talking into coconuts as if they were microphones to bring back western goods was the same thing as us making LLMs to recreate consciousness. They were using technology they understood to try to recreate something they didn't.
> Just because consciousness is something we don't fully understand doesn't mean it's the same thing as something else that's similar that we understand. That's a cargo-cult.
Nobody has said they’re aren’t different. And it’s not cargo cult to say you need philosophical conversations to understand how they’re different.
> The stone age Melanesian islanders making air strips out of bundles of weeds and talking into coconuts as if they were microphones to bring back western goods was the same thing as us making LLMs to recreate consciousness. They were using technology they understood to try to recreate something they didn't.
That’s not even remotely a true. Computer scientists understand how a neurone works. The conversations are around what constitutes as “intelligence”. That’s not a technical limitation of the models, it’s a philosophical problem we need to figure out.
No you don't understand how a neuron works fully or else you'd be able to make a synthetic one from chemical building blocks not using another living thing to do it. If you used another living thing to do it, you could say an ancient farmer understands plants cause they can put seeds in the ground. You have an incomplete model of it that is wrong but useful. Call me back when you can 3d-print a living person from raw chemical feedstock.
On top of all that, you're abusing the fallacy of composition. The bolt that is part of the aircraft carrier is light, therefore the aircraft carrier is also light. The atom is simple, therefore a living human being that is composed of atoms is also simple. Quantum mechanics is influenced by the observer, therefore you can change the external world just by thinking, and other new age stuff like that abuses a similar fallacy.
> No you don't understand how a neuron works fully or else you'd be able to make a synthetic one from chemical building blocks not using another living thing to do it.
We don’t know how a quantum mechanics works fully and yet we’ve built countless systems based on quantum behaviour including the CPUs that power the computer you’re replying to me on.
The point here is we’ve built something that emulates intelligence and thus further exploration into what intelligence is, is required. Arguing it isn’t intelligent because it was built differently isn’t an answer, it just dismisses the question entirely.
> On top of all that, you're abusing the fallacy of composition. The bolt that is part of the aircraft carrier is light, therefore the aircraft carrier is also light. The atom is simple, therefore a living human being that is composed of atoms is also simple. Quantum mechanics is influenced by the observer, therefore you can change the external world just by thinking, and other new age stuff like that abuses a similar fallacy.
You’re the one who keeps bringing up composition, not me. I’m the one telling you that you’re falling into that fallacy. So don’t start telling me that I’m wrong because of an argument you’re exclusively making.
My points were this:
1. we don’t have an understanding of what intelligence is and thus we need to explore that problem more deeply.
Given your comments about our ignorance, you of all people should agree with this point.
2. Arguing that AI is just “floating points” is overly reductive and misses the point that we don’t know what “intelligence” is.
So instead of claiming that AI is just numbers, a more constructive stance to take would be to build out better vocabulary and understanding about the distinctions between different types of “intelligence”.
Again, this aligns with your comment about the composition fallacy. Thus you if all people should be supporting my comments here.
Also your comment about us not understanding how a neurone works does vast injustice to the amount of research that has been done in this field. At this point in time we do have a very good understanding of how a neurone works. What we don’t understand is how consciousness arises from it. And that’s the very thing you seem to be arguing that there’s no point researching.
> Also your comment about us not understanding how a neurone works does vast injustice to the amount of research that has been done in this field. At this point in time we do have a very good understanding of how a neurone works. What we don’t understand is how consciousness arises from it. And that’s the very thing you seem to be arguing that there’s no point researching.
This is the part you keep stating that is wrong. We know a lot about the most obvious electrical way that neurons act. That is nowhere near having a good understanding of how they work.
Define “work”? Because the way they act is the way they work. What isn’t understood is how complex systems arise from these relatively simple parts. Which is the point I’ve made repeatedly.
I responded with an informative post but apparently made the mistake of disclosing it was generated and it was flagged. I reviewed the post and it is correct. If you cannot see it, it says you are wrong/incomplete here and cites three studies I know of:
Beniaguev, Segev & London, "Single cortical neurons as deep artificial neural networks," Neuron, 10 August 2021, Volume 109, Issue 17, pp. 2727–2739.e3
<a href="https://doi.org/10.1016/j.neuron.2021.07.002" rel="nofollow">https://doi.org/10.1016/j.neuron.2021.07.002
Open-access preprint: <a href="https://www.biorxiv.org/content/10.1101/613141v2" rel="nofollow">https://www.biorxiv.org/content/10.1101/613141v2
Gidon et al., "Dendritic action potentials and computation in human layer 2/3 cortical neurons," Science, 367(6473), 83–87 (2020)
<a href="https://doi.org/10.1126/science.aax6239" rel="nofollow">https://doi.org/10.1126/science.aax6239
Randi, Sharma, Dvali & Leifer, "Neural signal propagation atlas of Caenorhabditis elegans," Nature, 623(7986), 2023, pp. 406–414
<a href="https://doi.org/10.1038/s41586-023-06683-4" rel="nofollow">https://doi.org/10.1038/s41586-023-06683-4
Open-access preprint: <a href="https://arxiv.org/abs/2208.04790" rel="nofollow">https://arxiv.org/abs/2208.04790
Neurons are not individually simple, in any sense. Nor are they the only class of cell involved in cognitive processes — glial cells are involved in inhibition, memory, network modification, etc.
While I was not participating actively in the discussion - I just want to thank both participants for embodying the spirit of HN and academic discourse - especially in the face of a difficult debate that stirs strong emotions.
@svnt, @hnlmorg - this conversation - and the links it produced - are a fascinating read, and honestly gave me hope for discourse on the open internet.
> No you don't understand how a neuron works fully or else you'd be able to make a synthetic one from chemical building blocks not using another living thing to do it.
This is a bizarre contention. The test for whether something is understood isn't that we have the engineering capability to replicate it.
Just because you partially understand something like the ancient farmer partially understands plants by knowing to plant seeds, that doesn't mean you fully understand it and can thus reproduce all the properties of it synthetically like you are trying to reproduce a brain synthetically from scratch inside a computer with LLMs.
You can reproduce part of it, the intelligence, but consciousness and intelligence are not necessarily the same thing. This is why I compare it to cargo-culting. The Melanesian tribesmen thinks the coconut looks like a microphone, and he can use it as a paperweight just like a microphone and that's useful, but it is not actually a microphone.
A spider obviously "knows" how to build a spider web.
Does a spider "understand" how to build a spider web?
From a pragmatic theory of truth, a spider obviously "understands" how to build a spider web. A spider understands how to build a spider web much better than a human.
But when it comes to LLMs we start adding all this extra nonsense. As if the spider is blanket "smarter" than me because I can't spin a web. If the spider is "smarter" than me then it must also have this Cartesian theater of the mind like many people think they do.
It is layered philosophical misunderstandings to talk about LLMs in this way.
What could be an alternative explanation or approach for that spider? I’m out of loop, interested in what is currently academia trying to solve, thanks.
Neurons are different from GPU logic gates and the physical mechanisms by which a network of physical neurons computes answers is extremely different from the physical mechanism by which a GPU computes an answer.
> We need to stop thinking of LLMs as purely a piece of software
Intelligence from LLMs are an emergent property of probability. The LLM chooses a token to add to a chain of tokens based on the probability that humans will want to see that token. Humans see that token and assign it a qualitative but unfalsifiable intelligence score.
The first question we need to ask is whether a system whose claim of intelligence depends on the interpretation of what it produces by other intelligent beings, namely humans, intelligent or not. A cat is intelligent regardless of what humans think. Is an LLM intelligent regardless of what humans think?
If an LLM were to live a year with dolphins, would its weights change to respond to its new reality that it was now operating in dolphin-world? If you sent humans to live in dolphin world, we would eventually evolve to be less landloving and more dolphin like (Darwin thinks so, at least).
> If an LLM were to live a year with dolphins, would its weights change
Given three or more large scale training runs per year, probably yes once the training data is dominated by dolphin-related input. Much quicker in any case than the genes of the humans living with dolphins would be shaped by Evolution, which produces new generations much less frequently.
> Intelligence from LLMs are an emergent property of probability. The LLM chooses a token to add to a chain of tokens based on the probability that humans will want to see that token.
How is that any different to how our brains work?
> The first question we need to ask is whether a system whose claim of intelligence depends on the interpretation of what it produces by other intelligent beings, namely humans, intelligent or not.
Philosophically speaking, that’s also true of human intelligence. For example, the only evidence I have for your intelligence is what I ascribe from the text you’ve written.
> A cat is intelligent regardless of what humans think.
Are they? Humans can’t actually agree on whether cats are sentient. And often pet owners will be accused of anthropomorphising cats when describing intelligent behaviour in cats.
> Is an LLM intelligent regardless of what humans think?
That’s the question that needs researching.
> If an LLM were to live a year with dolphins, would its weights change to respond to its new reality that it was now operating in dolphin-world?
If it were trained on dolphin behaviour, then yes. Much like humans.
The difference here is that models don’t persist context in the same way humans do so the weights have to be hardcoded in a way that biological neurones are not.
> If you sent humans to live in dolphin world, we would eventually evolve to be less landloving and more dolphin like (Darwin thinks so, at least).
Evolution is an entirely different subject to intelligence. They’re not related in any way at all.
Plus humans have actually created new species of plants and animals. So it’s not like we can’t artificially put our thumb on the scale there either
———
I should add, I don’t actually think LLMs are sentient. I don’t even know if I’d call them “intelligent” (in the sense of what we describe in conversations such as this). But we also don’t know what intelligence is. Which is why we need to research this problem further.
Human neurons are neurons. Very similar to those neurons found in a fruit fly. A copy of those you can download to your local machine, right now, as a set of floating-point numbers. Then, you can (partially) simulate and make predictions of a fruit fly's neurons digitally, from those floating point numbers[1]. It even runs on your laptop.
That was in 2024. In Sep 2026, just a month ago, a group managed to map out an entire male fruit fly's brain for the first time[2]. That mapping can also be stored as a set of floating-point numbers. Mappings and simulations will continue to improve, as we've just recently started serious attempts at either.
So you have to believe one of three things. Either, human neurons cannot be simulated on a computer, ever, even though fly neurons can, even with our ability to simulate neurons digitally continuing to improve. Or, a simulation of a human brain, in the future, might think, talk, remember, all while lacking all consciousness. Or, a set of floating point numbers can be conscious (rather, a computation made with them).
No. The fruit fly connectome's simulation doesn't incorporate all the relevant mechanisms of the brain. There is more complexity in a single biological neuron than in a whole frontier LLM.
Sure it isn't. It's like taking a 2026 car and decrying all that complexity is necessary to haul cargo down a beaten path.
It's not. Two wheels and a box made out of whatever you have handy that's stiff enough is all you need. Everything else in a modern car is efficiency, safety, appeal to users, and a whole lot of details that only reflect our current economical and technological constraints of building things. 99% of the technical details of a car is about something else than the "primary goal".
Biology, even more so, is complex because of the way it's constructed (lots of tiny, independent, self-replicating machines) and how it adds up to a functioning organism.
You are missing the point. Biology is a kludge over billions of years and evolution is a blind process. It had many constraints that are not necessary for modern machines or systems. e.g. a GPU doesn't have to reproduce. Why didn't you drop yourself to planck scale or string theory scale and say we need all that detail to actually get a functioning AI.
Nor did I said it did. But either it's impossible to simulate their behaviour (regardless of how complex) in principle, ever, even in the far off future. Or it is possible. And what then? Is it a p-zombie?
Is a neuron arbitrarily complex ≠ is a neuron non-computable.
Of course we can't simulate a structure we don't even fully understand with full accuracy. But it's not magic. We will understand one day.
Chalmers used them as a thought experiment to argue against physicalism and never believed them to be real.
Similarly, if consciousness is something that neurons do, simulating them through floating point numbers on a computer should also bring it about. I think dismissing LLM consciousness because they're parameterised through floating point numbers is only possible if one hasn't thought it through long enough. Other arguments are much more reasonable / harder to object to.
Yes I roughly think the same. Brains are physical systems, as of yet we don't know anything beyond turing computability or physics in them, neural nets further are modeled close to neural structure, further complicating the matter. Under present knowledge, I think the existence of other conscious entities is quite possible because well we ourselves exist and are a physical system, I follow a form of Pascal's wager here. I see no cost to myself in not mistreating current AI's personally.
But if simulated neurons can create consciousness, there's no guarantee of what level of detail that simulation has to operate at in order to produce that effect.
For instance, it's (unlikely, but) theoretically possible that it would require simulating them down to the level of subatomic particles, with perfect accuracy.
Right now, we're simulating them at a much, much higher and more abstract level, and AFAIK we barely even have the processing power to do that for the time being. So an artificial consciousness from simulated neurons may (again, assuming it's possible—which is a decent assumption, but not a certainty) take decades or even more for us to be able to achieve.
The problem again is that (a) other forms of consciousness exist eg octopuses (b) I cannot get into the shoes of you or an LLM and see if I feel or think, so we only have external effects to go by. For things that look and seem so much like consciousness, I am rather cautious.
An octopus still has around 500 million neurons so that's not really a counterexample. Humans and octopuses are far more similar than humans and LLMs.
> But either it's impossible to simulate their behaviour (regardless of how complex) in principle, ever, even in the far off future. Or it is possible.
"Simulate" is not well defined (one can posit varying degrees of accuracy and complexity), but also there is the position that it doesn't matter. Simulating a thing doesn't create the thing.
The answer is yes, it is a p-zombie. What is so difficult about that?
Being able to make someone think a computer program is conscious does not make it conscious, because by that logic a Duke Nukem soundboard is conscious.
I am just shocked that anyone can be this overconfident in their view of the world and consciousness. It's frankly horrifying.
Do you genuinely think you've solved a problem that has confused humanity for centuries? Why don't you apply this brilliance elsewhere? Do we even need superintelligence if we have you?
You’re just making things up. A single neuron is about as complicated as an 8 layer neural network: <a href="https://www.sciencedirect.com/science/article/pii/S0896627321005018" rel="nofollow">https://www.sciencedirect.com/science/article/pii/S089662732...
Frontier LLMs are many orders of magnitude more complex. I don’t understand what it is about AI that just makes skeptics fabricate facts wholesale. There are plenty of factual arguments you could use.
I think your first option doesn’t necessarily have to distinguish between humans and non humans as a binary. Maybe at some level of complexity, neurons alone are not enough and the physical substrate comes into play. Penrose wrote a book where he makes the case that human intelligence cannot be purely computational and where he identifies the quantum nature of the brains’s microtubules as a possible mechanism.
Consciousness is itself is on a whole other level than just intelligence. I personally feel there has to be something metaphysical about it. Otherwise I think the only alternatives are super determinism and consciousness doesn’t actually exist, or numbers have consciousness.
My thoughts on this are a bit sloppy still, but here’s my view. I can’t rhyme the first alternative with ‘cogito ergo sum’ as to me it seems my thoughts have to exist somewhere. On the second alternative, this would imply nothing could actually exist but somehow there’s a principle that every possible decoding of an integer that represents something actually turns that something into existence. Where does that principle then come from and does that decoding not need something to run on?
I lean toward this seemingly absurd possibility, that either the phenomenon we call consciousness can arise from a cloud of floating point numbers (not even real, only computably so, lossy and noisy); and/or that numbers are already made of the same stuff of imagination as the mind. Not idealism, more like materialism where the "material" is naturally a medium of computation and thinking. The brain, the computer, the language models, our mathematics - what we call intelligence may not be so special or unique in the universe, but rather common and ubiquitous in nature. As if Turing completeness is trivial to accidentally achieve, it happens all the time. Oops, I think therefore I am.
You are massively over-stating the significance of the connectome. We've had the complete connectome (and lineage) for C.Elegans, with a hundredth the neurons and a 25-thousandth the synapses of a fruit fly, for decades and getting animal behaviours to emerge from simulations of it is a field very much still in its infancy.
Edit: You're also massively under-stating the significance of individual cell behaviour. I expect "[h]uman neurons are ... [v]ery similar to those neurons found in a fruit fly" would draw hollow laughter from any cellular neuroscientist. Here's a random overview of a bunch of recent papers on the importance of cellular diversity in one small part of a mammal brain: <a href="https://www.frontiersin.org/journals/cellular-neuroscience/articles/10.3389/fncel.2026.2010201/abstract" rel="nofollow">https://www.frontiersin.org/journals/cellular-neuroscience/a...
With that said, I've been thinking that consciousness is just mystery.
There's so much about the brain we don't know. Within that mystery, humans can read in all sorts of things. So we are conscious.
Machines are not a mystery. We made them, it's just numbers. They are not conscious.
If an AI can pull a confidence trick, I'm not sure what the qualitative difference is. "Chinese Room" problem, I guess? If the physical results are all we are able to compare, how do we define artificial vs real consciousness?
Atoms aren't a mystery. We know all the relevant laws of physics. No known law of physics could possibly lead to consciousness. Consciousness might as well be magic from the point of view of modern science.
Numbers versus atoms, what's the difference? Machines could certainly be conscious, by some weird, "magical" process. Maybe it is a crude, unrecognizable kind of consciousness, that turns on and off with the electricity.
But, given how we understand essentially nothing of where consciousness comes from, how can we possibly claim that it can never arise from just numbers?
It feels like this is preface for making one of two arguments: Either all living organisms are chattel, or no living organisms can be chattel. I say this be because saying everything is neurons (or whatever basic interconnect you choose) wipes away any demarcation one could use to differentiate between "has rights" and "does not have rights."
Here's the thing, trying to convince people that rely on something being chattel can only be resolved violently. We've run through this before, and know all the self-serving argument forms. They will latch on to their property until their hands are forcibly removed from it. It's part of why most of the money is on the pro-chattel side.
This seems like a rather fascile take, tbh. Along the lines of "everything is just atoms". Yes they're numbers. And yes, the hardware is just atoms. But its the emergent properties of those things that is significant, and the way that behaviour is embedded in human society - which can be just as morally consequential as swords and brains and harmful ideas.
thadt · · focus · HN ↗
> In a series of private meetings, the company consulted religious scholars to help instill morality into its FLOATING POINT NUMBERS — and make the case that their numbers could be conscious.
I recommend keeping a few NaNs in-hand if you’re worried. Spread a couple of around if your numbers start to stir and you’ll be right as rain.
> It appeared to Rabbi Navon that Mr. Olah and his team believed that FLOATING POINT NUMBERS had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
They’re cleverly designed and work well for their purpose, but can assure you they have neither dignity nor moral status.
> If the FLOATING POINT NUMBERS themselves could be made to choose goodness, Mr. Olah reasoned, the world would be more safe.
I get it. I’ve spent years trying to get my floating point numbers to choose goodness - I wish them luck.
Lerc · · focus · HN ↗
apsurd · · focus · HN ↗
It does seem exceptionally silly that a collection of floating point numbers has immediately outranked animals and other living beings. That’s why i like the parent comment. They’re numbers!
edit: maybe easier to say collections of atoms though are not to be treated equally. That’s reasonable isn’t it?
Lerc · · focus · HN ↗
Any determination is fundamentally flawed if it is derived from assuming that we know the absolute truth or falseness of the proposition that an algorithm can think.
It doesn't matter what you believe or feel or think, there is no consensus. If you are prepared to accept the belief that an individual can unilaterally decide the person-ness of something, then be prepared to accept when some individual decides to act on the principle that you don't have it.
novideonoradio · · focus · HN ↗
No, I will not be prepared to accept that, and neither will I accept that. This sounds like a threat, and a really funny one at that. You must accept that my floating numbers might be persons, otherwise... somebody will say that you're not! And then something bad might happen, who's to say.......
In the civilized world we all agree on the personhood of humans. If I say that your chatbot isn't a person, I am not uprooting the entire social understanding of personhood - I am not saying that suddenly you might not be a person, or that guy might not be a person. I'm saying that I refuse to consider a chatbot a person.
It's as if I were to come up to you with a brick in my hand, saying that this brick is a person. If you were to say "no", I would reply "well, if this brick isn't a person, your chatbot isn't a person either!"
How did you make such a big leap from consciousness to personhood anyway? They are two quite different things. Consciousness is a question of science, personhood is a question of culture. Pigs might be conscious and we treat them in a disgusting way, but even if we achieved a worldwide vegan utopia, wouldn't we still treat them differently from humans?
lazyasciiart · · focus · HN ↗
novideonoradio · · focus · HN ↗
My point is, if you go outside and look at a living human, you will go "yeah, that's a person", and so will any normal human.
foobar10000 · · focus · HN ↗
I take your point that a human walking around a city will - out of that 1000 responses - hopefully get 990+ of "yes, a person" - but that will start to degrade quickly once you hit boundaries.
And the less said about dehumanizaion - untermensh and that entire line of thinking - the better.
novideonoradio · · focus · HN ↗
[dead]
Lerc · · focus · HN ↗
You seem to have put a second tautology in to your tautology.
All right thinking people think rhat the things that qualify qualify.
If they didn't they wouldn't qualify, or be right thinking. But that's ok because we know the things that don't qualify don't qualify if we didn't know that we wouldn't be right thinking.
>How did you make such a big leap from consciousness to personhood anyway?
Once you start unilaterally making decisions about the unproven, you can decide all of the unprovables. History shows that personhood rolls into the firing range fairly quickly when people start doing that.
I'm more than comfortable saying you can't do that before someone openly declares that they are doing it.
novideonoradio · · focus · HN ↗
>Once you start unilaterally making decisions about the unproven, you can decide all of the unprovables.
You mean that if I say your dumbass chatbot isn't conscious, it's like a slippery slope and next thing you know you're not a person anymore, and you're in a concentration camp? Really, man?
hardbass · · focus · HN ↗
danaris · · focus · HN ↗
We understand consciousness well enough to say that a rock is not conscious.
We understand consciousness well enough to say that a human is conscious.
The existence of gray area does not mean that white and black do not exist.
We understand LLMs and consciousness well enough to say, at bare minimum, that it's incredibly unlikely that LLMs are conscious.
steve1977 · · focus · HN ↗
We treat other humans terribly too, so that doesn't seem to make a difference.
omnicognate · · focus · HN ↗
Lerc · · focus · HN ↗
ViscountPenguin · · focus · HN ↗
omnicognate · · focus · HN ↗
d1sxeyes · · focus · HN ↗
In the past, we’ve treated even other humans poorly because we think they’re different in meaningful ways, I sadly don’t see how this will be different with AI.
omnicognate · · focus · HN ↗
I think we have an extremely strong basis on which to say AI as it currently exists does not have conscious experiences, though, and if we start trying to pretend they do some real absurdities will result. Indeed the consequences could be pretty terrible in terms of power concentration, and I doubt it's a coincidence that those best placed to form a new priesthood of AI and tell everyone how it should and should not be treated are pushing the idea so strongly.
phoghed · · focus · HN ↗
Same for all the people who argue that they aren’t “intelligent” and don’t “know” things. Either they are/do, or that stuff isn’t as important as you’d think.
The only way it would matter is if you wanted to moralize about enslaving your computer or what it means to restart it.
omnicognate · · focus · HN ↗
That is precisely the context of the debate. You seem to find it absurd, as do I, but the reason we're talking about whether LLMs are conscious in this thread has nothing to do with whether they are useful. It's because important people, such as Dario Amodei, are engaging in exactly the kind of moralising over "AI welfare" that you're scoffing at.
phoghed · · focus · HN ↗
antonis-gr · · focus · HN ↗
aaron695 · · focus · HN ↗
[dead]
hnlmorg · · focus · HN ↗
What are biological neurones if not electric impulses of varying strengths?
We need to stop thinking of LLMs as purely a piece of software and ask those hard-to-define philosophical questions if we want any hope of building safety into our models.
narrator · · focus · HN ↗
hnlmorg · · focus · HN ↗
Nobody has said they’re aren’t different. And it’s not cargo cult to say you need philosophical conversations to understand how they’re different.
> The stone age Melanesian islanders making air strips out of bundles of weeds and talking into coconuts as if they were microphones to bring back western goods was the same thing as us making LLMs to recreate consciousness. They were using technology they understood to try to recreate something they didn't.
That’s not even remotely a true. Computer scientists understand how a neurone works. The conversations are around what constitutes as “intelligence”. That’s not a technical limitation of the models, it’s a philosophical problem we need to figure out.
narrator · · focus · HN ↗
On top of all that, you're abusing the fallacy of composition. The bolt that is part of the aircraft carrier is light, therefore the aircraft carrier is also light. The atom is simple, therefore a living human being that is composed of atoms is also simple. Quantum mechanics is influenced by the observer, therefore you can change the external world just by thinking, and other new age stuff like that abuses a similar fallacy.
[deleted] · · focus · HN ↗
[deleted]
hnlmorg · · focus · HN ↗
We don’t know how a quantum mechanics works fully and yet we’ve built countless systems based on quantum behaviour including the CPUs that power the computer you’re replying to me on.
The point here is we’ve built something that emulates intelligence and thus further exploration into what intelligence is, is required. Arguing it isn’t intelligent because it was built differently isn’t an answer, it just dismisses the question entirely.
> On top of all that, you're abusing the fallacy of composition. The bolt that is part of the aircraft carrier is light, therefore the aircraft carrier is also light. The atom is simple, therefore a living human being that is composed of atoms is also simple. Quantum mechanics is influenced by the observer, therefore you can change the external world just by thinking, and other new age stuff like that abuses a similar fallacy.
You’re the one who keeps bringing up composition, not me. I’m the one telling you that you’re falling into that fallacy. So don’t start telling me that I’m wrong because of an argument you’re exclusively making.
My points were this:
1. we don’t have an understanding of what intelligence is and thus we need to explore that problem more deeply.
Given your comments about our ignorance, you of all people should agree with this point.
2. Arguing that AI is just “floating points” is overly reductive and misses the point that we don’t know what “intelligence” is.
So instead of claiming that AI is just numbers, a more constructive stance to take would be to build out better vocabulary and understanding about the distinctions between different types of “intelligence”.
Again, this aligns with your comment about the composition fallacy. Thus you if all people should be supporting my comments here.
Also your comment about us not understanding how a neurone works does vast injustice to the amount of research that has been done in this field. At this point in time we do have a very good understanding of how a neurone works. What we don’t understand is how consciousness arises from it. And that’s the very thing you seem to be arguing that there’s no point researching.
svnt · · focus · HN ↗
This is the part you keep stating that is wrong. We know a lot about the most obvious electrical way that neurons act. That is nowhere near having a good understanding of how they work.
hnlmorg · · focus · HN ↗
svnt · · focus · HN ↗
[dead]
hnlmorg · · focus · HN ↗
I’m going to hold off any more constructive response because I’m not familiar with those research papers specifically so have some homework to do :)
But I just wanted you to be aware that your comment was appreciated even if HN flagged it unfairly.
(I can’t reply in thread for the same reason. But hopefully you see this).
svnt · · focus · HN ↗
Beniaguev, Segev & London, "Single cortical neurons as deep artificial neural networks," Neuron, 10 August 2021, Volume 109, Issue 17, pp. 2727–2739.e3 <a href="https://doi.org/10.1016/j.neuron.2021.07.002" rel="nofollow">https://doi.org/10.1016/j.neuron.2021.07.002 Open-access preprint: <a href="https://www.biorxiv.org/content/10.1101/613141v2" rel="nofollow">https://www.biorxiv.org/content/10.1101/613141v2
Gidon et al., "Dendritic action potentials and computation in human layer 2/3 cortical neurons," Science, 367(6473), 83–87 (2020) <a href="https://doi.org/10.1126/science.aax6239" rel="nofollow">https://doi.org/10.1126/science.aax6239
Randi, Sharma, Dvali & Leifer, "Neural signal propagation atlas of Caenorhabditis elegans," Nature, 623(7986), 2023, pp. 406–414 <a href="https://doi.org/10.1038/s41586-023-06683-4" rel="nofollow">https://doi.org/10.1038/s41586-023-06683-4 Open-access preprint: <a href="https://arxiv.org/abs/2208.04790" rel="nofollow">https://arxiv.org/abs/2208.04790
Neurons are not individually simple, in any sense. Nor are they the only class of cell involved in cognitive processes — glial cells are involved in inhibition, memory, network modification, etc.
foobar10000 · · focus · HN ↗
@svnt, @hnlmorg - this conversation - and the links it produced - are a fascinating read, and honestly gave me hope for discourse on the open internet.
squidbeak · · focus · HN ↗
This is a bizarre contention. The test for whether something is understood isn't that we have the engineering capability to replicate it.
narrator · · focus · HN ↗
You can reproduce part of it, the intelligence, but consciousness and intelligence are not necessarily the same thing. This is why I compare it to cargo-culting. The Melanesian tribesmen thinks the coconut looks like a microphone, and he can use it as a paperweight just like a microphone and that's useful, but it is not actually a microphone.
hnlmorg · · focus · HN ↗
The crux of the matter seems to be that you think it isn’t worth our effort researching just because it isn’t identical.
But that isn’t how research and discovery happens.
lutanista · · focus · HN ↗
A spider obviously "knows" how to build a spider web.
Does a spider "understand" how to build a spider web?
From a pragmatic theory of truth, a spider obviously "understands" how to build a spider web. A spider understands how to build a spider web much better than a human.
But when it comes to LLMs we start adding all this extra nonsense. As if the spider is blanket "smarter" than me because I can't spin a web. If the spider is "smarter" than me then it must also have this Cartesian theater of the mind like many people think they do.
It is layered philosophical misunderstandings to talk about LLMs in this way.
BOOSTERHIDROGEN · · focus · HN ↗
irenaeus · · focus · HN ↗
[deleted] · · focus · HN ↗
[deleted]
hnlmorg · · focus · HN ↗
You shouldn’t compare biological neurones to microprocessor logic gates because that’s like comparing a car to a pair of shoes.
saimiam · · focus · HN ↗
Intelligence from LLMs are an emergent property of probability. The LLM chooses a token to add to a chain of tokens based on the probability that humans will want to see that token. Humans see that token and assign it a qualitative but unfalsifiable intelligence score.
The first question we need to ask is whether a system whose claim of intelligence depends on the interpretation of what it produces by other intelligent beings, namely humans, intelligent or not. A cat is intelligent regardless of what humans think. Is an LLM intelligent regardless of what humans think?
If an LLM were to live a year with dolphins, would its weights change to respond to its new reality that it was now operating in dolphin-world? If you sent humans to live in dolphin world, we would eventually evolve to be less landloving and more dolphin like (Darwin thinks so, at least).
mofeien · · focus · HN ↗
Given three or more large scale training runs per year, probably yes once the training data is dominated by dolphin-related input. Much quicker in any case than the genes of the humans living with dolphins would be shaped by Evolution, which produces new generations much less frequently.
hnlmorg · · focus · HN ↗
How is that any different to how our brains work?
> The first question we need to ask is whether a system whose claim of intelligence depends on the interpretation of what it produces by other intelligent beings, namely humans, intelligent or not.
Philosophically speaking, that’s also true of human intelligence. For example, the only evidence I have for your intelligence is what I ascribe from the text you’ve written.
> A cat is intelligent regardless of what humans think.
Are they? Humans can’t actually agree on whether cats are sentient. And often pet owners will be accused of anthropomorphising cats when describing intelligent behaviour in cats.
> Is an LLM intelligent regardless of what humans think?
That’s the question that needs researching.
> If an LLM were to live a year with dolphins, would its weights change to respond to its new reality that it was now operating in dolphin-world?
If it were trained on dolphin behaviour, then yes. Much like humans.
The difference here is that models don’t persist context in the same way humans do so the weights have to be hardcoded in a way that biological neurones are not.
> If you sent humans to live in dolphin world, we would eventually evolve to be less landloving and more dolphin like (Darwin thinks so, at least).
Evolution is an entirely different subject to intelligence. They’re not related in any way at all.
Plus humans have actually created new species of plants and animals. So it’s not like we can’t artificially put our thumb on the scale there either
———
I should add, I don’t actually think LLMs are sentient. I don’t even know if I’d call them “intelligent” (in the sense of what we describe in conversations such as this). But we also don’t know what intelligence is. Which is why we need to research this problem further.
phi0 · · focus · HN ↗
That was in 2024. In Sep 2026, just a month ago, a group managed to map out an entire male fruit fly's brain for the first time[2]. That mapping can also be stored as a set of floating-point numbers. Mappings and simulations will continue to improve, as we've just recently started serious attempts at either.
So you have to believe one of three things. Either, human neurons cannot be simulated on a computer, ever, even though fly neurons can, even with our ability to simulate neurons digitally continuing to improve. Or, a simulation of a human brain, in the future, might think, talk, remember, all while lacking all consciousness. Or, a set of floating point numbers can be conscious (rather, a computation made with them).
[1]: <a href="https://www.nature.com/articles/s41586-024-07763-9" rel="nofollow">https://www.nature.com/articles/s41586-024-07763-9 [2]: <a href="https://research.google/blog/a-connectomics-milestone-mapping-the-complete-male-fruit-fly-brain/" rel="nofollow">https://research.google/blog/a-connectomics-milestone-mappin...
EMIRELADERO · · focus · HN ↗
conmod278 · · focus · HN ↗
TeMPOraL · · focus · HN ↗
It's not. Two wheels and a box made out of whatever you have handy that's stiff enough is all you need. Everything else in a modern car is efficiency, safety, appeal to users, and a whole lot of details that only reflect our current economical and technological constraints of building things. 99% of the technical details of a car is about something else than the "primary goal".
Biology, even more so, is complex because of the way it's constructed (lots of tiny, independent, self-replicating machines) and how it adds up to a functioning organism.
conmod278 · · focus · HN ↗
zombot · · focus · HN ↗
TeMPOraL · · focus · HN ↗
I see absolutely no reason to believe human mind cannot be ultimately boiled down to finite number of floats and run on future commodity hardware.
phi0 · · focus · HN ↗
Is a neuron arbitrarily complex ≠ is a neuron non-computable.
Of course we can't simulate a structure we don't even fully understand with full accuracy. But it's not magic. We will understand one day.
hardbass · · focus · HN ↗
phi0 · · focus · HN ↗
Similarly, if consciousness is something that neurons do, simulating them through floating point numbers on a computer should also bring it about. I think dismissing LLM consciousness because they're parameterised through floating point numbers is only possible if one hasn't thought it through long enough. Other arguments are much more reasonable / harder to object to.
hardbass · · focus · HN ↗
danaris · · focus · HN ↗
For instance, it's (unlikely, but) theoretically possible that it would require simulating them down to the level of subatomic particles, with perfect accuracy.
Right now, we're simulating them at a much, much higher and more abstract level, and AFAIK we barely even have the processing power to do that for the time being. So an artificial consciousness from simulated neurons may (again, assuming it's possible—which is a decent assumption, but not a certainty) take decades or even more for us to be able to achieve.
hardbass · · focus · HN ↗
qlte · · focus · HN ↗
zahlman · · focus · HN ↗
"Simulate" is not well defined (one can posit varying degrees of accuracy and complexity), but also there is the position that it doesn't matter. Simulating a thing doesn't create the thing.
irenaeus · · focus · HN ↗
Being able to make someone think a computer program is conscious does not make it conscious, because by that logic a Duke Nukem soundboard is conscious.
solenoid0937 · · focus · HN ↗
Do you genuinely think you've solved a problem that has confused humanity for centuries? Why don't you apply this brilliance elsewhere? Do we even need superintelligence if we have you?
Some epistemic humility is warranted.
cindyllm · · focus · HN ↗
[dead]
zombot · · focus · HN ↗
topicalSoup · · focus · HN ↗
Frontier LLMs are many orders of magnitude more complex. I don’t understand what it is about AI that just makes skeptics fabricate facts wholesale. There are plenty of factual arguments you could use.
geertj · · focus · HN ↗
I think your first option doesn’t necessarily have to distinguish between humans and non humans as a binary. Maybe at some level of complexity, neurons alone are not enough and the physical substrate comes into play. Penrose wrote a book where he makes the case that human intelligence cannot be purely computational and where he identifies the quantum nature of the brains’s microtubules as a possible mechanism.
Consciousness is itself is on a whole other level than just intelligence. I personally feel there has to be something metaphysical about it. Otherwise I think the only alternatives are super determinism and consciousness doesn’t actually exist, or numbers have consciousness.
My thoughts on this are a bit sloppy still, but here’s my view. I can’t rhyme the first alternative with ‘cogito ergo sum’ as to me it seems my thoughts have to exist somewhere. On the second alternative, this would imply nothing could actually exist but somehow there’s a principle that every possible decoding of an integer that represents something actually turns that something into existence. Where does that principle then come from and does that decoding not need something to run on?
lioeters · · focus · HN ↗
I lean toward this seemingly absurd possibility, that either the phenomenon we call consciousness can arise from a cloud of floating point numbers (not even real, only computably so, lossy and noisy); and/or that numbers are already made of the same stuff of imagination as the mind. Not idealism, more like materialism where the "material" is naturally a medium of computation and thinking. The brain, the computer, the language models, our mathematics - what we call intelligence may not be so special or unique in the universe, but rather common and ubiquitous in nature. As if Turing completeness is trivial to accidentally achieve, it happens all the time. Oops, I think therefore I am.
omnicognate · · focus · HN ↗
Edit: You're also massively under-stating the significance of individual cell behaviour. I expect "[h]uman neurons are ... [v]ery similar to those neurons found in a fruit fly" would draw hollow laughter from any cellular neuroscientist. Here's a random overview of a bunch of recent papers on the importance of cellular diversity in one small part of a mammal brain: <a href="https://www.frontiersin.org/journals/cellular-neuroscience/articles/10.3389/fncel.2026.2010201/abstract" rel="nofollow">https://www.frontiersin.org/journals/cellular-neuroscience/a...
Andrex · · focus · HN ↗
With that said, I've been thinking that consciousness is just mystery.
There's so much about the brain we don't know. Within that mystery, humans can read in all sorts of things. So we are conscious.
Machines are not a mystery. We made them, it's just numbers. They are not conscious.
If an AI can pull a confidence trick, I'm not sure what the qualitative difference is. "Chinese Room" problem, I guess? If the physical results are all we are able to compare, how do we define artificial vs real consciousness?
greeneggs · · focus · HN ↗
Numbers versus atoms, what's the difference? Machines could certainly be conscious, by some weird, "magical" process. Maybe it is a crude, unrecognizable kind of consciousness, that turns on and off with the electricity.
But, given how we understand essentially nothing of where consciousness comes from, how can we possibly claim that it can never arise from just numbers?
SquibblesRedux · · focus · HN ↗
salawat · · focus · HN ↗
SequoiaHope · · focus · HN ↗
999900000999 · · focus · HN ↗
How many of us work for absolutely amoral companies because we need to eat.
If you’ve ever been in a situation where people are really struggling you’ll see that morality fades away very real quick.
No Gods, no souls, we’re just biological machines driven to eat and mate.
andyjohnson0 · · focus · HN ↗
This seems like a rather fascile take, tbh. Along the lines of "everything is just atoms". Yes they're numbers. And yes, the hardware is just atoms. But its the emergent properties of those things that is significant, and the way that behaviour is embedded in human society - which can be just as morally consequential as swords and brains and harmful ideas.
[deleted] · · focus · HN ↗
[deleted]
mrshadowgoose · · focus · HN ↗