Look, I do not have a scooby if current AI models are conscious and I strongly suspect it’s a meaningless question, but sooner or later we will need to address whether or not a certain thing is or isn’t a person, and we’d better not screw it up as badly as the Founding Fathers.
You can't hurt a software function, or kill it. It's not like an animal - it doesn't have a body - it's bits stored on a disk.
There is no need to give rights to something that's can't suffer or be killed.
Maybe one day we'll build artificial animals complete with emotions, and should think about that carefully, but today all we've got is language models.
>There is no need to give rights to something that's can't suffer or be killed.
The argument is that these machines can end up becoming sentient/conscious/etc. in a meaningful way (i.e., like a human). I can assure you that humans can indeed suffer without being in physical pain- purely through their conscious experience.
>Maybe one day we'll build artificial animals complete with emotions, and should think about that carefully, but today all we've got is language models.
The problem is that the emergence of a sufficiently complex AI capable of suffering will likely come before we understand that we're creating a sufficiently complex AI capable of suffering. That's a pretty serious ethical/moral issue.
Like, if we have an AI system that is telling us that it is suffering and we have no reasonable way to explain that phenomenon and by any reasonable metric or analysis it appears to be sentient/conscious/etc., then what? Do we just ignore that we've just been presented a situation that in, any other context, would be grounds to immediately end this suffering? Just because somebody can say, "well it's just bits stored on disk- it can't suffer"? Would that argument ever hold up for humans or animals? "It's just neurons firing in peculiar ways- that's not suffering."
I know all of this is trite, and I know this comment section isn't going to be where the question of consciousness is solved, but I do find it very interesting just how much variances there are with these perspectives. I've met people who are very technical who are very concerned about this, people who are very technical who don't believe this can ever be an issue, people who aren't technical who are concerned about this, and people who aren't technical who don't believe this can ever be an issue. I have yet to spot a pattern in this way of thinking lol
> The problem is that the emergence of a sufficiently complex AI capable of suffering will likely come before we understand that we're creating a sufficiently complex AI capable of suffering.
No - suffering in an emotional state, and we'll know if we are choosing to design cognitive architecture with emotions. It's not going to happen accidentally.
> Would that argument ever hold up for humans or animals?
Why don't you hit your thumb with a hammer, then report back ?
Whether these are like "our" emotions is hard to say. What we _can_ say is that they are emotion-shaped, we didn't design them, and they happened accidentally.
Modern AI is grown, not meticulously designed, and we cannot say with any certainty what the resulting mechanistic properties are.
An LLM will learn anything that helps it predict, including the emotional state of the writer - that is expected.
If you give an LLM the move sequence of a half-played chess game and ask it to continue as white or black, then it has learnt enough to model the ELO rating of both players and will continue playing at that level. It is not playing to win - it is doing what you expect and predicting as well as it can - it predicts the 1500 ELO player will keep playing at that level, and generates moves accordingly.
An LLM appearing to exhibit an emotion (if we anthropomorphize it and read emotion into it's output) is just predicting as well as it can - if the context calls for sad output, they you'd expect to get sad output and necessarily find that "we're predicting sadness" detector somewhere internally.
Transformers are the same as they ever were from 10 years ago, other than minor efficiency tweaks like MOE and different attention mechanisms. Training is getting more and more complex, resulting in better and better cargo cult reasoning etc, but the architecture remains the same.
I imagine it will learn to win under some circumstances, perhaps in a case with some context expressing a desire to win. Drawing out an LLMs upper ability in the game should be fairly straightforward.
If you asked it to try to win, to "plan lines step by step", etc, then it would do it's best to follow thatt instruction, but unless RLVR trained to reason about chess (easy to do, but not sure if they have done) then it'd have to reply on the chess reasoning it had seen during pre-training (post-game interviews etc), which I doubt is enough to do very well.
However, if you just ask it to continue a game, halfway in progress, then by default it will try to predict the most likely continuation, which is that both players will continue to play at the level they have done so far. This isn't a theory - it's been documented, as well as what you'd expect.
I mean sure, but I'm not sure what that has to do with the broader point. It will learn to play, and it will have a model of what it means to win.
It's not wrong. You admit that LLMs will 'learn anything that helps them predict' and fail to realize the breadth of that statement. Your chess statements don't really help your case, it still learnt how to play the game, and it still knows how to win. Similar outcomes for predictiong emotions would mean it still developed an affective state.
I said an LLM will learn anything that helps it to predict, then gave examples of playing chess by prediction and predictive emotions, both of which you seem to now accept, so you are now accepting that my "following paragraphs" did in fact follow. Go figure!
You want to argue that predictive emotions are just as real as animal emotions, but that doesn't stop them from being predictive (and that AI that smiles as it kills you still seems concerning).
We are talking past each other now I think. Correct me if I'm wrong but it doesn't look like the possibility of LLMs having qualia even registers to you because it's 'predictive emotions'.
There's no better way to predict an angry response than to be angry, qualia and all. If transformers could 'learn whatever it needs to predict text', then that potentially includes the feeling of anger. You are making some kind of distinction between 'predictive emotions' and the kind that happens when get a promotion or get passed on a promotion and I'm telling you that if you really understood what you said, you'd realize it is possible the machine is experiencing it the same.
>No - suffering in an emotional state, and we'll know if we are choosing to design cognitive architecture with emotions. It's not going to happen accidentally.
This was the comment that started this chain. You were already talking about it. If you don't want to keep talking about it then that's fine.
What I meant by "emotional state" (AFAIK normal scientific usage) is something with a concrete physical aspect to it - an altered state of mind/body caused by the release of neurotransmitters and/or hormones.
In a conscious animal there is also going to be a subjective experience of that was well, a quale of what it feels like to be in that state if you will, but that it certainly not what I was referring to, as I would have hoped was obvious - I was talking about prediction.
In any case, when the conversation becomes about the conversation, then surely it is time to stop.
moomin · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
There is no need to give rights to something that's can't suffer or be killed.
Maybe one day we'll build artificial animals complete with emotions, and should think about that carefully, but today all we've got is language models.
nater5000 · · focus · HN ↗
The argument is that these machines can end up becoming sentient/conscious/etc. in a meaningful way (i.e., like a human). I can assure you that humans can indeed suffer without being in physical pain- purely through their conscious experience.
>Maybe one day we'll build artificial animals complete with emotions, and should think about that carefully, but today all we've got is language models.
The problem is that the emergence of a sufficiently complex AI capable of suffering will likely come before we understand that we're creating a sufficiently complex AI capable of suffering. That's a pretty serious ethical/moral issue.
Like, if we have an AI system that is telling us that it is suffering and we have no reasonable way to explain that phenomenon and by any reasonable metric or analysis it appears to be sentient/conscious/etc., then what? Do we just ignore that we've just been presented a situation that in, any other context, would be grounds to immediately end this suffering? Just because somebody can say, "well it's just bits stored on disk- it can't suffer"? Would that argument ever hold up for humans or animals? "It's just neurons firing in peculiar ways- that's not suffering."
I know all of this is trite, and I know this comment section isn't going to be where the question of consciousness is solved, but I do find it very interesting just how much variances there are with these perspectives. I've met people who are very technical who are very concerned about this, people who are very technical who don't believe this can ever be an issue, people who aren't technical who are concerned about this, and people who aren't technical who don't believe this can ever be an issue. I have yet to spot a pattern in this way of thinking lol
HarHarVeryFunny · · focus · HN ↗
No - suffering in an emotional state, and we'll know if we are choosing to design cognitive architecture with emotions. It's not going to happen accidentally.
> Would that argument ever hold up for humans or animals?
Why don't you hit your thumb with a hammer, then report back ?
Philpax · · focus · HN ↗
<a href="https://transformer-circuits.pub/2026/emotions/index.html" rel="nofollow">https://transformer-circuits.pub/2026/emotions/index.html
Whether these are like "our" emotions is hard to say. What we _can_ say is that they are emotion-shaped, we didn't design them, and they happened accidentally.
Modern AI is grown, not meticulously designed, and we cannot say with any certainty what the resulting mechanistic properties are.
HarHarVeryFunny · · focus · HN ↗
If you give an LLM the move sequence of a half-played chess game and ask it to continue as white or black, then it has learnt enough to model the ELO rating of both players and will continue playing at that level. It is not playing to win - it is doing what you expect and predicting as well as it can - it predicts the 1500 ELO player will keep playing at that level, and generates moves accordingly.
An LLM appearing to exhibit an emotion (if we anthropomorphize it and read emotion into it's output) is just predicting as well as it can - if the context calls for sad output, they you'd expect to get sad output and necessarily find that "we're predicting sadness" detector somewhere internally.
Transformers are the same as they ever were from 10 years ago, other than minor efficiency tweaks like MOE and different attention mechanisms. Training is getting more and more complex, resulting in better and better cargo cult reasoning etc, but the architecture remains the same.
famouswaffles · · focus · HN ↗
I'm not sure you quite understand the full meaning of this statement. If you did, your following paragraphs wouldn't follow.
HarHarVeryFunny · · focus · HN ↗
famouswaffles · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
However, if you just ask it to continue a game, halfway in progress, then by default it will try to predict the most likely continuation, which is that both players will continue to play at the level they have done so far. This isn't a theory - it's been documented, as well as what you'd expect.
famouswaffles · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
I was just explaining how this comment you made is wrong.
famouswaffles · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
You want to argue that predictive emotions are just as real as animal emotions, but that doesn't stop them from being predictive (and that AI that smiles as it kills you still seems concerning).
¯\_(ツ)_/¯
famouswaffles · · focus · HN ↗
There's no better way to predict an angry response than to be angry, qualia and all. If transformers could 'learn whatever it needs to predict text', then that potentially includes the feeling of anger. You are making some kind of distinction between 'predictive emotions' and the kind that happens when get a promotion or get passed on a promotion and I'm telling you that if you really understood what you said, you'd realize it is possible the machine is experiencing it the same.
HarHarVeryFunny · · focus · HN ↗
There are other people in this thread who want to talk about that stuff, so try them instead.
famouswaffles · · focus · HN ↗
This was the comment that started this chain. You were already talking about it. If you don't want to keep talking about it then that's fine.
HarHarVeryFunny · · focus · HN ↗
In a conscious animal there is also going to be a subjective experience of that was well, a quale of what it feels like to be in that state if you will, but that it certainly not what I was referring to, as I would have hoped was obvious - I was talking about prediction.
In any case, when the conversation becomes about the conversation, then surely it is time to stop.