When thinking about ‘AI’ I sometimes find it useful to replace the terms ‘model’ or ‘AI’ or whatever with a description of what we’re really talking about here: a whole bunch of floating point numbers.
> In a series of private meetings, the company consulted religious scholars to help instill morality into its FLOATING POINT NUMBERS — and make the case that their numbers could be conscious.
I recommend keeping a few NaNs in-hand if you’re worried. Spread a couple of around if your numbers start to stir and you’ll be right as rain.
> It appeared to Rabbi Navon that Mr. Olah and his team believed that FLOATING POINT NUMBERS had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
They’re cleverly designed and work well for their purpose, but can assure you they have neither dignity nor moral status.
> If the FLOATING POINT NUMBERS themselves could be made to choose goodness, Mr. Olah reasoned, the world would be more safe.
I get it. I’ve spent years trying to get my floating point numbers to choose goodness - I wish them luck.
That’s overly reductive and thus completely misses the problem entirely.
What are biological neurones if not electric impulses of varying strengths?
We need to stop thinking of LLMs as purely a piece of software and ask those hard-to-define philosophical questions if we want any hope of building safety into our models.
> We need to stop thinking of LLMs as purely a piece of software
Intelligence from LLMs are an emergent property of probability. The LLM chooses a token to add to a chain of tokens based on the probability that humans will want to see that token. Humans see that token and assign it a qualitative but unfalsifiable intelligence score.
The first question we need to ask is whether a system whose claim of intelligence depends on the interpretation of what it produces by other intelligent beings, namely humans, intelligent or not. A cat is intelligent regardless of what humans think. Is an LLM intelligent regardless of what humans think?
If an LLM were to live a year with dolphins, would its weights change to respond to its new reality that it was now operating in dolphin-world? If you sent humans to live in dolphin world, we would eventually evolve to be less landloving and more dolphin like (Darwin thinks so, at least).
> Intelligence from LLMs are an emergent property of probability. The LLM chooses a token to add to a chain of tokens based on the probability that humans will want to see that token.
How is that any different to how our brains work?
> The first question we need to ask is whether a system whose claim of intelligence depends on the interpretation of what it produces by other intelligent beings, namely humans, intelligent or not.
Philosophically speaking, that’s also true of human intelligence. For example, the only evidence I have for your intelligence is what I ascribe from the text you’ve written.
> A cat is intelligent regardless of what humans think.
Are they? Humans can’t actually agree on whether cats are sentient. And often pet owners will be accused of anthropomorphising cats when describing intelligent behaviour in cats.
> Is an LLM intelligent regardless of what humans think?
That’s the question that needs researching.
> If an LLM were to live a year with dolphins, would its weights change to respond to its new reality that it was now operating in dolphin-world?
If it were trained on dolphin behaviour, then yes. Much like humans.
The difference here is that models don’t persist context in the same way humans do so the weights have to be hardcoded in a way that biological neurones are not.
> If you sent humans to live in dolphin world, we would eventually evolve to be less landloving and more dolphin like (Darwin thinks so, at least).
Evolution is an entirely different subject to intelligence. They’re not related in any way at all.
Plus humans have actually created new species of plants and animals. So it’s not like we can’t artificially put our thumb on the scale there either
———
I should add, I don’t actually think LLMs are sentient. I don’t even know if I’d call them “intelligent” (in the sense of what we describe in conversations such as this). But we also don’t know what intelligence is. Which is why we need to research this problem further.
thadt · · focus · HN ↗
> In a series of private meetings, the company consulted religious scholars to help instill morality into its FLOATING POINT NUMBERS — and make the case that their numbers could be conscious.
I recommend keeping a few NaNs in-hand if you’re worried. Spread a couple of around if your numbers start to stir and you’ll be right as rain.
> It appeared to Rabbi Navon that Mr. Olah and his team believed that FLOATING POINT NUMBERS had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
They’re cleverly designed and work well for their purpose, but can assure you they have neither dignity nor moral status.
> If the FLOATING POINT NUMBERS themselves could be made to choose goodness, Mr. Olah reasoned, the world would be more safe.
I get it. I’ve spent years trying to get my floating point numbers to choose goodness - I wish them luck.
hnlmorg · · focus · HN ↗
What are biological neurones if not electric impulses of varying strengths?
We need to stop thinking of LLMs as purely a piece of software and ask those hard-to-define philosophical questions if we want any hope of building safety into our models.
saimiam · · focus · HN ↗
Intelligence from LLMs are an emergent property of probability. The LLM chooses a token to add to a chain of tokens based on the probability that humans will want to see that token. Humans see that token and assign it a qualitative but unfalsifiable intelligence score.
The first question we need to ask is whether a system whose claim of intelligence depends on the interpretation of what it produces by other intelligent beings, namely humans, intelligent or not. A cat is intelligent regardless of what humans think. Is an LLM intelligent regardless of what humans think?
If an LLM were to live a year with dolphins, would its weights change to respond to its new reality that it was now operating in dolphin-world? If you sent humans to live in dolphin world, we would eventually evolve to be less landloving and more dolphin like (Darwin thinks so, at least).
hnlmorg · · focus · HN ↗
How is that any different to how our brains work?
> The first question we need to ask is whether a system whose claim of intelligence depends on the interpretation of what it produces by other intelligent beings, namely humans, intelligent or not.
Philosophically speaking, that’s also true of human intelligence. For example, the only evidence I have for your intelligence is what I ascribe from the text you’ve written.
> A cat is intelligent regardless of what humans think.
Are they? Humans can’t actually agree on whether cats are sentient. And often pet owners will be accused of anthropomorphising cats when describing intelligent behaviour in cats.
> Is an LLM intelligent regardless of what humans think?
That’s the question that needs researching.
> If an LLM were to live a year with dolphins, would its weights change to respond to its new reality that it was now operating in dolphin-world?
If it were trained on dolphin behaviour, then yes. Much like humans.
The difference here is that models don’t persist context in the same way humans do so the weights have to be hardcoded in a way that biological neurones are not.
> If you sent humans to live in dolphin world, we would eventually evolve to be less landloving and more dolphin like (Darwin thinks so, at least).
Evolution is an entirely different subject to intelligence. They’re not related in any way at all.
Plus humans have actually created new species of plants and animals. So it’s not like we can’t artificially put our thumb on the scale there either
———
I should add, I don’t actually think LLMs are sentient. I don’t even know if I’d call them “intelligent” (in the sense of what we describe in conversations such as this). But we also don’t know what intelligence is. Which is why we need to research this problem further.