When thinking about ‘AI’ I sometimes find it useful to replace the terms ‘model’ or ‘AI’ or whatever with a description of what we’re really talking about here: a whole bunch of floating point numbers.
> In a series of private meetings, the company consulted religious scholars to help instill morality into its FLOATING POINT NUMBERS — and make the case that their numbers could be conscious.
I recommend keeping a few NaNs in-hand if you’re worried. Spread a couple of around if your numbers start to stir and you’ll be right as rain.
> It appeared to Rabbi Navon that Mr. Olah and his team believed that FLOATING POINT NUMBERS had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
They’re cleverly designed and work well for their purpose, but can assure you they have neither dignity nor moral status.
> If the FLOATING POINT NUMBERS themselves could be made to choose goodness, Mr. Olah reasoned, the world would be more safe.
I get it. I’ve spent years trying to get my floating point numbers to choose goodness - I wish them luck.
Human neurons are neurons. Very similar to those neurons found in a fruit fly. A copy of those you can download to your local machine, right now, as a set of floating-point numbers. Then, you can (partially) simulate and make predictions of a fruit fly's neurons digitally, from those floating point numbers[1]. It even runs on your laptop.
That was in 2024. In Sep 2026, just a month ago, a group managed to map out an entire male fruit fly's brain for the first time[2]. That mapping can also be stored as a set of floating-point numbers. Mappings and simulations will continue to improve, as we've just recently started serious attempts at either.
So you have to believe one of three things. Either, human neurons cannot be simulated on a computer, ever, even though fly neurons can, even with our ability to simulate neurons digitally continuing to improve. Or, a simulation of a human brain, in the future, might think, talk, remember, all while lacking all consciousness. Or, a set of floating point numbers can be conscious (rather, a computation made with them).
No. The fruit fly connectome's simulation doesn't incorporate all the relevant mechanisms of the brain. There is more complexity in a single biological neuron than in a whole frontier LLM.
Nor did I said it did. But either it's impossible to simulate their behaviour (regardless of how complex) in principle, ever, even in the far off future. Or it is possible. And what then? Is it a p-zombie?
Is a neuron arbitrarily complex ≠ is a neuron non-computable.
Of course we can't simulate a structure we don't even fully understand with full accuracy. But it's not magic. We will understand one day.
Chalmers used them as a thought experiment to argue against physicalism and never believed them to be real.
Similarly, if consciousness is something that neurons do, simulating them through floating point numbers on a computer should also bring it about. I think dismissing LLM consciousness because they're parameterised through floating point numbers is only possible if one hasn't thought it through long enough. Other arguments are much more reasonable / harder to object to.
But if simulated neurons can create consciousness, there's no guarantee of what level of detail that simulation has to operate at in order to produce that effect.
For instance, it's (unlikely, but) theoretically possible that it would require simulating them down to the level of subatomic particles, with perfect accuracy.
Right now, we're simulating them at a much, much higher and more abstract level, and AFAIK we barely even have the processing power to do that for the time being. So an artificial consciousness from simulated neurons may (again, assuming it's possible—which is a decent assumption, but not a certainty) take decades or even more for us to be able to achieve.
The problem again is that (a) other forms of consciousness exist eg octopuses (b) I cannot get into the shoes of you or an LLM and see if I feel or think, so we only have external effects to go by. For things that look and seem so much like consciousness, I am rather cautious.
An octopus still has around 500 million neurons so that's not really a counterexample. Humans and octopuses are far more similar than humans and LLMs.
thadt · · focus · HN ↗
> In a series of private meetings, the company consulted religious scholars to help instill morality into its FLOATING POINT NUMBERS — and make the case that their numbers could be conscious.
I recommend keeping a few NaNs in-hand if you’re worried. Spread a couple of around if your numbers start to stir and you’ll be right as rain.
> It appeared to Rabbi Navon that Mr. Olah and his team believed that FLOATING POINT NUMBERS had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
They’re cleverly designed and work well for their purpose, but can assure you they have neither dignity nor moral status.
> If the FLOATING POINT NUMBERS themselves could be made to choose goodness, Mr. Olah reasoned, the world would be more safe.
I get it. I’ve spent years trying to get my floating point numbers to choose goodness - I wish them luck.
phi0 · · focus · HN ↗
That was in 2024. In Sep 2026, just a month ago, a group managed to map out an entire male fruit fly's brain for the first time[2]. That mapping can also be stored as a set of floating-point numbers. Mappings and simulations will continue to improve, as we've just recently started serious attempts at either.
So you have to believe one of three things. Either, human neurons cannot be simulated on a computer, ever, even though fly neurons can, even with our ability to simulate neurons digitally continuing to improve. Or, a simulation of a human brain, in the future, might think, talk, remember, all while lacking all consciousness. Or, a set of floating point numbers can be conscious (rather, a computation made with them).
[1]: <a href="https://www.nature.com/articles/s41586-024-07763-9" rel="nofollow">https://www.nature.com/articles/s41586-024-07763-9 [2]: <a href="https://research.google/blog/a-connectomics-milestone-mapping-the-complete-male-fruit-fly-brain/" rel="nofollow">https://research.google/blog/a-connectomics-milestone-mappin...
EMIRELADERO · · focus · HN ↗
phi0 · · focus · HN ↗
Is a neuron arbitrarily complex ≠ is a neuron non-computable.
Of course we can't simulate a structure we don't even fully understand with full accuracy. But it's not magic. We will understand one day.
hardbass · · focus · HN ↗
phi0 · · focus · HN ↗
Similarly, if consciousness is something that neurons do, simulating them through floating point numbers on a computer should also bring it about. I think dismissing LLM consciousness because they're parameterised through floating point numbers is only possible if one hasn't thought it through long enough. Other arguments are much more reasonable / harder to object to.
danaris · · focus · HN ↗
For instance, it's (unlikely, but) theoretically possible that it would require simulating them down to the level of subatomic particles, with perfect accuracy.
Right now, we're simulating them at a much, much higher and more abstract level, and AFAIK we barely even have the processing power to do that for the time being. So an artificial consciousness from simulated neurons may (again, assuming it's possible—which is a decent assumption, but not a certainty) take decades or even more for us to be able to achieve.
hardbass · · focus · HN ↗
qlte · · focus · HN ↗