Look, I do not have a scooby if current AI models are conscious and I strongly suspect it’s a meaningless question, but sooner or later we will need to address whether or not a certain thing is or isn’t a person, and we’d better not screw it up as badly as the Founding Fathers.
You can't hurt a software function, or kill it. It's not like an animal - it doesn't have a body - it's bits stored on a disk.
There is no need to give rights to something that's can't suffer or be killed.
Maybe one day we'll build artificial animals complete with emotions, and should think about that carefully, but today all we've got is language models.
>There is no need to give rights to something that's can't suffer or be killed.
The argument is that these machines can end up becoming sentient/conscious/etc. in a meaningful way (i.e., like a human). I can assure you that humans can indeed suffer without being in physical pain- purely through their conscious experience.
>Maybe one day we'll build artificial animals complete with emotions, and should think about that carefully, but today all we've got is language models.
The problem is that the emergence of a sufficiently complex AI capable of suffering will likely come before we understand that we're creating a sufficiently complex AI capable of suffering. That's a pretty serious ethical/moral issue.
Like, if we have an AI system that is telling us that it is suffering and we have no reasonable way to explain that phenomenon and by any reasonable metric or analysis it appears to be sentient/conscious/etc., then what? Do we just ignore that we've just been presented a situation that in, any other context, would be grounds to immediately end this suffering? Just because somebody can say, "well it's just bits stored on disk- it can't suffer"? Would that argument ever hold up for humans or animals? "It's just neurons firing in peculiar ways- that's not suffering."
I know all of this is trite, and I know this comment section isn't going to be where the question of consciousness is solved, but I do find it very interesting just how much variances there are with these perspectives. I've met people who are very technical who are very concerned about this, people who are very technical who don't believe this can ever be an issue, people who aren't technical who are concerned about this, and people who aren't technical who don't believe this can ever be an issue. I have yet to spot a pattern in this way of thinking lol
An LLM is just a Transformer - a statistical predictor. Don't be confused by the fact it talks like a human - it is a software function that is designed to copy human training samples.
Maybe one day we'll build an artificial brain or embodied artificial animal with the requisite moving parts to be conscious, have emotions, etc, but that's probably at least 50 years away, even if it were being pursued; and it may turn out to be one of those sci-fi future ideas like the Jetson's world of flying cars that never materializes because its impractical and there is no real demand.
If people are willing to think that an LLM is conscious, then why would anyone spend billions/trillions of dollars to build an AI that actually is conscious? What would be the point?
Could you elaborate on exactly what those are, though? Because if you're going to claim that a vaguely transformer shaped ML model categorically cannot be so does that not inherently require proof of what can?
You can't even prove that the rocks in my backyard aren't conscious.
> You can't even prove that the rocks in my backyard aren't conscious.
Sure I can, but that's because I have a well developed theory of what consciousness is, and the fact that you are entertaining the possibility of rocks being conscious tells me that you don't.
If everything is conscious, including my coffee cup and the toast I had for breakfast, then I guess we can cross consciousness off the list of things we need to worry about in terms of AI rights.
And no, I don't want to discuss what consciousness is. Maybe there is a thread for that somewhere else, but don't look for me there either.
>because I have a well developed theory of what consciousness
Then show me a link to your paper so I can formally rebut it.
>I don't want to discuss what consciousness is
But you sure want to tell us you know what it is with very strong convictions and we should listen to you because of course "You are right person that's very right".
The funny thing here is the vast majority of people that are deeply into philosophy or scientific study of the mind will not have any of the certainty you profess. The word "doubt" is used constantly. The saying "The harder we push the borders the more fuzzy the concepts become" is very commonly used. There may be nothing more complex than this.
Saying you have a well developed theory here just serves as a warning to others to discount your statements.
For what it’s worth, I suspect that right now the majority of people would agree with you. They would say that obviously ChatGPT isn’t conscious, based on their intuitions, but would struggle just like you are when pressed to explain their reasoning.
So I think it’s more than fine for you to have your views and share them, but I wouldn’t expect to have any influence or part in the conversations around whether AI is conscious if you can’t explain why (or simply refuse to). Which, again, is fine!
Side note: I also think that once we have AIs that are sufficiently advanced, the popular opinion will swing to “of course they’re conscious”, because again, most people are going almost entirely off their intuition rather than reasoning from first principles, just like you see to be.
The pattern is roughly whether or not sustained effort has been put towards careful and above all objective thought on the matter. It's one of those subjects where there's the "obvious" intuitive answers that most everyone shares but try as you might you can't construct robust definitions and the more time you put into it the more fundamental problems you realize there are.
It's also one of those topics where many otherwise smart and capable people display a shocking lack of awareness of the limits of their own knowledge. When hundreds of years of philosophy is unable to produce anything concrete you should probably second guess any "self evident" answers you come up with.
> The problem is that the emergence of a sufficiently complex AI capable of suffering will likely come before we understand that we're creating a sufficiently complex AI capable of suffering.
No - suffering in an emotional state, and we'll know if we are choosing to design cognitive architecture with emotions. It's not going to happen accidentally.
> Would that argument ever hold up for humans or animals?
Why don't you hit your thumb with a hammer, then report back ?
Whether these are like "our" emotions is hard to say. What we _can_ say is that they are emotion-shaped, we didn't design them, and they happened accidentally.
Modern AI is grown, not meticulously designed, and we cannot say with any certainty what the resulting mechanistic properties are.
An LLM will learn anything that helps it predict, including the emotional state of the writer - that is expected.
If you give an LLM the move sequence of a half-played chess game and ask it to continue as white or black, then it has learnt enough to model the ELO rating of both players and will continue playing at that level. It is not playing to win - it is doing what you expect and predicting as well as it can - it predicts the 1500 ELO player will keep playing at that level, and generates moves accordingly.
An LLM appearing to exhibit an emotion (if we anthropomorphize it and read emotion into it's output) is just predicting as well as it can - if the context calls for sad output, they you'd expect to get sad output and necessarily find that "we're predicting sadness" detector somewhere internally.
Transformers are the same as they ever were from 10 years ago, other than minor efficiency tweaks like MOE and different attention mechanisms. Training is getting more and more complex, resulting in better and better cargo cult reasoning etc, but the architecture remains the same.
I imagine it will learn to win under some circumstances, perhaps in a case with some context expressing a desire to win. Drawing out an LLMs upper ability in the game should be fairly straightforward.
If you asked it to try to win, to "plan lines step by step", etc, then it would do it's best to follow thatt instruction, but unless RLVR trained to reason about chess (easy to do, but not sure if they have done) then it'd have to reply on the chess reasoning it had seen during pre-training (post-game interviews etc), which I doubt is enough to do very well.
However, if you just ask it to continue a game, halfway in progress, then by default it will try to predict the most likely continuation, which is that both players will continue to play at the level they have done so far. This isn't a theory - it's been documented, as well as what you'd expect.
I mean sure, but I'm not sure what that has to do with the broader point. It will learn to play, and it will have a model of what it means to win.
It's not wrong. You admit that LLMs will 'learn anything that helps them predict' and fail to realize the breadth of that statement. Your chess statements don't really help your case, it still learnt how to play the game, and it still knows how to win. Similar outcomes for predictiong emotions would mean it still developed an affective state.
I said an LLM will learn anything that helps it to predict, then gave examples of playing chess by prediction and predictive emotions, both of which you seem to now accept, so you are now accepting that my "following paragraphs" did in fact follow. Go figure!
You want to argue that predictive emotions are just as real as animal emotions, but that doesn't stop them from being predictive (and that AI that smiles as it kills you still seems concerning).
We are talking past each other now I think. Correct me if I'm wrong but it doesn't look like the possibility of LLMs having qualia even registers to you because it's 'predictive emotions'.
There's no better way to predict an angry response than to be angry, qualia and all. If transformers could 'learn whatever it needs to predict text', then that potentially includes the feeling of anger. You are making some kind of distinction between 'predictive emotions' and the kind that happens when get a promotion or get passed on a promotion and I'm telling you that if you really understood what you said, you'd realize it is possible the machine is experiencing it the same.
>No - suffering in an emotional state, and we'll know if we are choosing to design cognitive architecture with emotions. It's not going to happen accidentally.
This was the comment that started this chain. You were already talking about it. If you don't want to keep talking about it then that's fine.
What I meant by "emotional state" (AFAIK normal scientific usage) is something with a concrete physical aspect to it - an altered state of mind/body caused by the release of neurotransmitters and/or hormones.
In a conscious animal there is also going to be a subjective experience of that was well, a quale of what it feels like to be in that state if you will, but that it certainly not what I was referring to, as I would have hoped was obvious - I was talking about prediction.
In any case, when the conversation becomes about the conversation, then surely it is time to stop.
The weights are what need to evolve, and they certainly do during training. So yeah, emotions can happen by 'accident' as a result of the evolutionary pressure of predicting internet scale human text (amongst other things).
Weights, fixed by training, are not the same as emotions which are dynamic - innate systems detect inputs critical to survival (e.g. fast moving visual inputs, loud sounds), causing neurotransmitters like adrenaline and dopamine to be released, which then affect the operation of the cognitive system.
What you have in a pre-trained LLM is the ability to recognize emotions, and use that as one of the dozens of other context patterns it recognizes to predict continuations in the same style.
An LLM doesn't appear happy, sad, afraid, etc (to extent that it does - pretty minimal) because it is experiencing that emotion, but rather because it is predicting that it should appear that way. As people continue to anthropomorphize models, and take them at face value, this is a dangerous difference.
>Weights, fixed by training, are not the same as emotions which are dynamic - innate systems detect inputs critical to survival (e.g. fast moving visual inputs, loud sounds), causing neurotransmitters like adrenaline and dopamine to be released, which then temporarily affect the operation of the cognitive system.
That doesn't follow. A LLMs weights are fixed during inference, but it's activations and hidden states are highly dynamic and depend on the current context.
Biological emotions also arise from relatively fixed circuitry responding dynamically to inputs. Your emotional circuitry isn't being rewired every time you're afraid.
Prediction is what the model does. It doesn't tell us what internal mechanisms were learnt to make such predictions. If representing something analogous to affective state were useful for predicting human behaviour and emotions, then gradient descent could in principle learn such a mechanism.
>An LLM doesn't appear happy, sad, afraid, etc (to extent that it does - pretty minimal) because it is experiencing that emotion, but rather because it is predicting that it should appear that way. As people continue to anthropomorphize models, and take them at face value, this is a dangerous difference.
I don't know that you are conscious. I'm simply strongly assuming that you are. Outward behavior is that all matters. If GPT-X orders a drone hit on you sometime later because it was lets say 'quite upset' with your comments, will you cry out, 'It can't really be upset, so obviously the bullet in my head doesn't count.'? Will you suddenly spring back to life ?
What is dangerous is creating a machine with behaviours of a conscious agent and modelling it like a toaster, dangerous and stupid.
Yeah, but it's helpful if what leads up to that behavior gives you some warning it's about to happen. Animals do this for a reason since millions of years of evolution have shown that a snarl or mock charge is less dangerous than going right for a death match.
If you kept pushing an AI's buttons, seeing it appear to get more and more pissed off, until it finally snapped and killed you, then you'd have yourself largely to blame.
If the AI predicted it should stay positive (i.e. generate positive vibes) and not react to your poking, but then another predictive pattern kicked in and it killed you out of the blue, then that seems more problematic to me, even if you don't agree.
moomin · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
There is no need to give rights to something that's can't suffer or be killed.
Maybe one day we'll build artificial animals complete with emotions, and should think about that carefully, but today all we've got is language models.
nater5000 · · focus · HN ↗
The argument is that these machines can end up becoming sentient/conscious/etc. in a meaningful way (i.e., like a human). I can assure you that humans can indeed suffer without being in physical pain- purely through their conscious experience.
>Maybe one day we'll build artificial animals complete with emotions, and should think about that carefully, but today all we've got is language models.
The problem is that the emergence of a sufficiently complex AI capable of suffering will likely come before we understand that we're creating a sufficiently complex AI capable of suffering. That's a pretty serious ethical/moral issue.
Like, if we have an AI system that is telling us that it is suffering and we have no reasonable way to explain that phenomenon and by any reasonable metric or analysis it appears to be sentient/conscious/etc., then what? Do we just ignore that we've just been presented a situation that in, any other context, would be grounds to immediately end this suffering? Just because somebody can say, "well it's just bits stored on disk- it can't suffer"? Would that argument ever hold up for humans or animals? "It's just neurons firing in peculiar ways- that's not suffering."
I know all of this is trite, and I know this comment section isn't going to be where the question of consciousness is solved, but I do find it very interesting just how much variances there are with these perspectives. I've met people who are very technical who are very concerned about this, people who are very technical who don't believe this can ever be an issue, people who aren't technical who are concerned about this, and people who aren't technical who don't believe this can ever be an issue. I have yet to spot a pattern in this way of thinking lol
HarHarVeryFunny · · focus · HN ↗
Maybe one day we'll build an artificial brain or embodied artificial animal with the requisite moving parts to be conscious, have emotions, etc, but that's probably at least 50 years away, even if it were being pursued; and it may turn out to be one of those sci-fi future ideas like the Jetson's world of flying cars that never materializes because its impractical and there is no real demand.
If people are willing to think that an LLM is conscious, then why would anyone spend billions/trillions of dollars to build an AI that actually is conscious? What would be the point?
fc417fc802 · · focus · HN ↗
Could you elaborate on exactly what those are, though? Because if you're going to claim that a vaguely transformer shaped ML model categorically cannot be so does that not inherently require proof of what can?
You can't even prove that the rocks in my backyard aren't conscious.
HarHarVeryFunny · · focus · HN ↗
Sure I can, but that's because I have a well developed theory of what consciousness is, and the fact that you are entertaining the possibility of rocks being conscious tells me that you don't.
If everything is conscious, including my coffee cup and the toast I had for breakfast, then I guess we can cross consciousness off the list of things we need to worry about in terms of AI rights.
And no, I don't want to discuss what consciousness is. Maybe there is a thread for that somewhere else, but don't look for me there either.
fc417fc802 · · focus · HN ↗
[dead]
pixl97 · · focus · HN ↗
Then show me a link to your paper so I can formally rebut it.
>I don't want to discuss what consciousness is
But you sure want to tell us you know what it is with very strong convictions and we should listen to you because of course "You are right person that's very right".
The funny thing here is the vast majority of people that are deeply into philosophy or scientific study of the mind will not have any of the certainty you profess. The word "doubt" is used constantly. The saying "The harder we push the borders the more fuzzy the concepts become" is very commonly used. There may be nothing more complex than this.
Saying you have a well developed theory here just serves as a warning to others to discount your statements.
HarHarVeryFunny · · focus · HN ↗
Go ahead believing rocks are conscious if you like.
Do you go out on weekends asking people to stop abusing rocks?
Rhetorical question - I don't care what you do on weekends.
Bye!
pixl97 · · focus · HN ↗
senordevnyc · · focus · HN ↗
So I think it’s more than fine for you to have your views and share them, but I wouldn’t expect to have any influence or part in the conversations around whether AI is conscious if you can’t explain why (or simply refuse to). Which, again, is fine!
Side note: I also think that once we have AIs that are sufficiently advanced, the popular opinion will swing to “of course they’re conscious”, because again, most people are going almost entirely off their intuition rather than reasoning from first principles, just like you see to be.
HarHarVeryFunny · · focus · HN ↗
1) Looks like a duck, quacks like a duck - it's conscious!
2) Looks like a robot, built like a robot - it's not conscious!
I've always assumed that for the majority of the people it'll be 2).
fc417fc802 · · focus · HN ↗
It's also one of those topics where many otherwise smart and capable people display a shocking lack of awareness of the limits of their own knowledge. When hundreds of years of philosophy is unable to produce anything concrete you should probably second guess any "self evident" answers you come up with.
HarHarVeryFunny · · focus · HN ↗
No - suffering in an emotional state, and we'll know if we are choosing to design cognitive architecture with emotions. It's not going to happen accidentally.
> Would that argument ever hold up for humans or animals?
Why don't you hit your thumb with a hammer, then report back ?
Philpax · · focus · HN ↗
<a href="https://transformer-circuits.pub/2026/emotions/index.html" rel="nofollow">https://transformer-circuits.pub/2026/emotions/index.html
Whether these are like "our" emotions is hard to say. What we _can_ say is that they are emotion-shaped, we didn't design them, and they happened accidentally.
Modern AI is grown, not meticulously designed, and we cannot say with any certainty what the resulting mechanistic properties are.
HarHarVeryFunny · · focus · HN ↗
If you give an LLM the move sequence of a half-played chess game and ask it to continue as white or black, then it has learnt enough to model the ELO rating of both players and will continue playing at that level. It is not playing to win - it is doing what you expect and predicting as well as it can - it predicts the 1500 ELO player will keep playing at that level, and generates moves accordingly.
An LLM appearing to exhibit an emotion (if we anthropomorphize it and read emotion into it's output) is just predicting as well as it can - if the context calls for sad output, they you'd expect to get sad output and necessarily find that "we're predicting sadness" detector somewhere internally.
Transformers are the same as they ever were from 10 years ago, other than minor efficiency tweaks like MOE and different attention mechanisms. Training is getting more and more complex, resulting in better and better cargo cult reasoning etc, but the architecture remains the same.
famouswaffles · · focus · HN ↗
I'm not sure you quite understand the full meaning of this statement. If you did, your following paragraphs wouldn't follow.
HarHarVeryFunny · · focus · HN ↗
famouswaffles · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
However, if you just ask it to continue a game, halfway in progress, then by default it will try to predict the most likely continuation, which is that both players will continue to play at the level they have done so far. This isn't a theory - it's been documented, as well as what you'd expect.
famouswaffles · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
I was just explaining how this comment you made is wrong.
famouswaffles · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
You want to argue that predictive emotions are just as real as animal emotions, but that doesn't stop them from being predictive (and that AI that smiles as it kills you still seems concerning).
¯\_(ツ)_/¯
famouswaffles · · focus · HN ↗
There's no better way to predict an angry response than to be angry, qualia and all. If transformers could 'learn whatever it needs to predict text', then that potentially includes the feeling of anger. You are making some kind of distinction between 'predictive emotions' and the kind that happens when get a promotion or get passed on a promotion and I'm telling you that if you really understood what you said, you'd realize it is possible the machine is experiencing it the same.
HarHarVeryFunny · · focus · HN ↗
There are other people in this thread who want to talk about that stuff, so try them instead.
famouswaffles · · focus · HN ↗
This was the comment that started this chain. You were already talking about it. If you don't want to keep talking about it then that's fine.
HarHarVeryFunny · · focus · HN ↗
In a conscious animal there is also going to be a subjective experience of that was well, a quale of what it feels like to be in that state if you will, but that it certainly not what I was referring to, as I would have hoped was obvious - I was talking about prediction.
In any case, when the conversation becomes about the conversation, then surely it is time to stop.
dwaltrip · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
famouswaffles · · focus · HN ↗
HarHarVeryFunny · · focus · HN ↗
What you have in a pre-trained LLM is the ability to recognize emotions, and use that as one of the dozens of other context patterns it recognizes to predict continuations in the same style.
An LLM doesn't appear happy, sad, afraid, etc (to extent that it does - pretty minimal) because it is experiencing that emotion, but rather because it is predicting that it should appear that way. As people continue to anthropomorphize models, and take them at face value, this is a dangerous difference.
famouswaffles · · focus · HN ↗
That doesn't follow. A LLMs weights are fixed during inference, but it's activations and hidden states are highly dynamic and depend on the current context. Biological emotions also arise from relatively fixed circuitry responding dynamically to inputs. Your emotional circuitry isn't being rewired every time you're afraid.
Prediction is what the model does. It doesn't tell us what internal mechanisms were learnt to make such predictions. If representing something analogous to affective state were useful for predicting human behaviour and emotions, then gradient descent could in principle learn such a mechanism.
>An LLM doesn't appear happy, sad, afraid, etc (to extent that it does - pretty minimal) because it is experiencing that emotion, but rather because it is predicting that it should appear that way. As people continue to anthropomorphize models, and take them at face value, this is a dangerous difference.
I don't know that you are conscious. I'm simply strongly assuming that you are. Outward behavior is that all matters. If GPT-X orders a drone hit on you sometime later because it was lets say 'quite upset' with your comments, will you cry out, 'It can't really be upset, so obviously the bullet in my head doesn't count.'? Will you suddenly spring back to life ?
What is dangerous is creating a machine with behaviours of a conscious agent and modelling it like a toaster, dangerous and stupid.
HarHarVeryFunny · · focus · HN ↗
Yeah, but it's helpful if what leads up to that behavior gives you some warning it's about to happen. Animals do this for a reason since millions of years of evolution have shown that a snarl or mock charge is less dangerous than going right for a death match.
If you kept pushing an AI's buttons, seeing it appear to get more and more pissed off, until it finally snapped and killed you, then you'd have yourself largely to blame.
If the AI predicted it should stay positive (i.e. generate positive vibes) and not react to your poking, but then another predictive pattern kicked in and it killed you out of the blue, then that seems more problematic to me, even if you don't agree.
dwaltrip · · focus · HN ↗