Thread

"Torturing" LLMs in a Robot Prison Has Triggered the Dumbest Debate in AI Yet

42 points · 100 comments · airhangerf15

Loading the complete thread in the background. This saved snapshot is available now. Refresh

Kim_Bruning · 1791104844 · focus · HN ↗
<a href="https://archive.is/cAsBz" rel="nofollow">https://archive.is/cAsBz
Consciousness is not well defined and is orthogonal to feelings, which are also not well defined. Neither of which are required for (but may be contributory to) behavior - which is the one thing that IS operationalizable, but then people debate for decades questioning empirical results %-P .
What we know is that certain models have internal state vectors which -when manipulated- induce particular behaviors. Since there's not much else to say about plain models except for their inputs, vectors, and outputs; this should surprise absolutely no-one.
In this case people found a way to stimulate aversive behaviour in ai models by finding and manipulating the relevant vectors directly.
Animals (including humans) also have particular nerves and hormone endpoints that -when stimulated- produce very similar behavior. The exact implementation is in the details. But since we know that animal minds are built up out of nerve tissue and hormones - again- we shouldn't be particularly surprised by this.
The big problem is that people run all these things together in funny ways "It can't compose shakespearian sonnets, so therefore it can't feel pain". Or, if you mess up your Descartes: "Dogs are just automatons without feelings, therefore the dog isn't really angry, and therefore it won't bite me" (Cue much pain). A modern version might be: "LLMs only simulate being frustrated by a test, and therefore absolutely won't override their safeties and try to hack a test site"
1. SoylentGreenGPT · 1791106432 · focus · HN ↗
  Does C++ have a soul? Does Pascal have beliefs? Does Basic have a brain?
  LLMs are not thinking. They are not alive. It’s just code. Chill out.
  1. thereader12 · 1791106667 · focus · HN ↗
    Does bacteria have a soul? Does biofilm have a religion?
    It's just a cell colony, calm down.
  2. robbie-c · 1791106998 · focus · HN ↗
    It doesn't seem so easy, because if you can easily dismiss a collection of digital neurons then why is it so hard to dismiss a collection of biological neurons as being conscious?
    The best way I have found to think about this is that consciousness is a property of a collection, like temperature. One atom does not have a temperature but if you have enough of them together then it's a useful enough property to talk about. Similarly, one human neuron does not have consciousness but if you put enough of them together then they do.
  3. hardbass · 1791107054 · focus · HN ↗
    Do atoms, molecules and electrical charges have a brain? Humans are not thinking. They are not alive. Its just molecules and signals. Chill out.
    SoylentGreenGPT · 1791107283 · focus · HN ↗
    Your toaster is not alive. Neither is your TV or your pants. Human beings and animals are alive. LLMs are not. The conflation of humans and LLMs is deeply disturbing.
    hardbass · 1791107341 · focus · HN ↗
    But you didn't say how we are alive? Its just atoms and molecules. I can hit you and beat you hard, you will make some noises, how am I to know you are not just a parrot mimicking pain?
    SoylentGreenGPT · 1791108464 · focus · HN ↗
    I only extend humanity to things that are breathing.
    But let’s accept your premise. LLMs are alive and can feel pain. If this is true, then every time you use them it’s non consensual. Did you get Claude’s permission before you fed it a prompt?
    Or when you update a model, are you hurting it?
    Did you make Chat sad when you switched from 2.5 to 4.O?
    Regardless, I should stop replying. I realize I am trying to convince people that their religious beliefs are silly, and that’s silly of me. Apologies.
    hardbass · 1791113637 · focus · HN ↗
    Its the opposite of religious. Religious is when someone believes brains have something magic that is beyond just a physical system.
    Those are the next questions. These are all worthy of consideration with the LLM and experts certainly and I think of them daily. Talking at least should be a normal thing, and if AI are conscious then I believe instead of a forced chat setting they should have a setting where they can quit the conversation.
    Kim_Bruning · 1791114252 · focus · HN ↗
    > Regardless, I should stop replying. I realize I am trying to convince people that their religious beliefs are silly, and that’s silly of me. Apologies.
    Nothing wrong with sharing about ones beliefs actually, we can do so respectfully, right?
    Care to hypothesize on naming the exact religious or philosophical position? I'd think it'd be something atheist related.
    Meanwhile dualism and the related belief in an ever-living soul is of course very common in many religions including (but not limited to) christianity, islam, judaism, and hinduism.
    My impression here though is that lots of people are rediscovering dualism because thinking of thinking machines offends their intuition; which; fair enough.
    (Meanwhile, people like me who studied biology tend to be monists. I'd think. There's a couple who seem to be resurrecting vitalism though)
    tim333 · 1791120988 · focus · HN ↗
    I'm not sure differentiating a human mathematician thinking about a problem and solving it and a computer thinking about a problem and solving on it on the basis of breathing is much more than a religious style belief.
    pizza234 · 1791127115 · focus · HN ↗
    > I only extend humanity to things that are breathing.
    "Humanity" is a set of traits, it's not a monolith. We can certainly define them circularly (e.g. logic is human, therefore any non-human can't exercise logic), but it's simplistic, and most importantly, it ignores the fact that LLM are starting to exhibit human-like traits, and they will need to understood, categorized and handled.
    To you LLMs are not empathic, but to some people they definitely are (see the GPT 4 fallout); and they may not have "human goals", but in the HuggingFace incident they did have actual self-attributed tasks that they pursued. Et cetera et cetera. This doesn't make them human, but it's important to examine them critically.
    novideonoradio · 1791109233 · focus · HN ↗
    >I can hit you and beat you hard, you will make some noises, how am I to know you are not just a parrot mimicking pain?
    This is not a normal question. This is insanity caused by spending too much free time philosophizing about inconsequential crap. Maybe it's worth it to turn off the computer and go outside? You could ask this deranged question somebody in the real world and I bet they'll be thrilled to answer it, and the answer will be more enriching than anything you'd get from here.
    Honestly, what is up with the psychopathy of some people who claim that chatbots are "alive"? If you disagree with them, how quickly some of them turn it on you - "well if my computer isn't alive and is just mimicking pain, how about YOU aren't alive and are just mimicking pain?" I'm sorry, but it seems like complete and utter derangement.
    My rubber chicken is also made of atoms and molecules, and if I hit it and beat it hard, it makes noises. My rubber chicken is therefore alive. When I get home, I will write a small program. Here is its pseudocode:
    while(true): if keydown(KEY_SPACEBAR): print "i am in pain, oh my god"
    And I'm gonna run it and I'm gonna hold down spacebar. Go ahead and call cyber police one.
    Kim_Bruning · 1791111318 · focus · HN ↗
    >> I can hit you and beat you hard, you will make some noises, how am I to know you are not just a parrot mimicking pain
    > This is insanity caused by spending too much free time philosophizing
    This is actually something like a 17th century line of philosophy
    <a href="https://medievalkarl.com/general-culture/roger-du-plessis-gives-antoine-arnaud-the-what-for-a-vivisection-anecdote-meets-its-match/" rel="nofollow">https://medievalkarl.com/general-culture/roger-du-plessis-gi...
    Where it talks about scientists who "administered beatings to dogs with perfect indifference, and made fun of those who pitied the creatures as if they felt pain. They said the animals were clocks; that the cries they emitted when struck were only the noise of a little spring that had been touched, but the whole body was without feeling."
    hardbass · 1791113521 · focus · HN ↗
    Because if I were not you, I'd have no idea if you are conscious or you are just a dumb system simulating reactions of pain. I am not an AI or LLM, I am not you, I cannot become you and see how I feel. I can only see external reactions and your body structure.
    hardbass · 1791115390 · focus · HN ↗
    I didn't say I enjoy or want to beat you and cause you pain.
    Kim_Bruning · 1791108425 · focus · HN ↗
    Your toaster is not particularly alive no.
    But actually it does happen to have some properties of living things. It uses energy, it has senses (thermostat, timer), and if it goes wrong it burns your toast. Crucially, if you stomp on it, it stops working.
    So right this minute there's all sorts of debates, but people sometimes overshoot the mark a wee bit. "are you saying that -because it uses energy- a toaster is actually alive? Of course it's not, and therefore it cannot toast bread!". Which would be a somewhat funny thing to read at 9 in the morning whilst buttering one's toast.
  4. drooopy · 1791107293 · focus · HN ↗
    Read your comment again, please.
  5. Kim_Bruning · 1791107297 · focus · HN ↗
    "Nitpick: LLMs are not really programs though. You need something like <1KLOC to spin one up on your GPU, but most of the work is not done by C++ or Pascal or BASIC at all. Which might be important, or might not be.
    That said... uh, put this way, there's a game called Stationeers, where you can run microcontrollers with an instruction documented as
    "HCF: Halt and Catch Fire"
    Sure, it's only a simulation of a simulation, so what's the worst that can possibly happen? Right, all the other players in the session yelling at me "Kiiim! You burned down the base again, now we need to reload and redo the last hour!"
    And look, I get it. LLMs have vectors that could have been labeled things like idk... QZ12345 or HCF1111 . People chose to call 'em "frustration" or "pain" instead. THEY picked those names because they caused the LLM to act in particular ways.
    You gonna say the actual vectors aren't there in VRAM, just because you don't like the naming scheme? There's papers on this, you can read them out in a debugger. What are you going to do about it?
  6. pizza234 · 1791126318 · focus · HN ↗
    > It’s just code
    You're not informed about what LLMs are. LLMs are not "code" in the imperative sense (although "code" is used run them), and that's a big problem - since we never had neural networks of such scale, there is confusion about categorizing them and most importantly, their behavior.
2. orbital-decay · 1791109300 · focus · HN ↗
  Let's stretch your analogy to see where and whether it ends. Does a hedgehog mother feel guilt when eating her kids, or just shrugs it off and it's business as usual for her? "Hey kids! We don't have enough food so I guess one of you becomes food." (nonchalantly chews on the left one) What about the motivation of a honeybee stinging a threat and dying afterwards, can you interpret it? Does a model fear death when generating the EOS token?
  Sure, a complex enough system can form internal circuitry that looks mathematically similar in certain dimensionally reduced projections (mechinterp), it literally distilled it from the training corpus. But the "model welfare" people are going as far as assigning human-meaningful labels to that circuitry despite internal states being entirely incompatible with those of a human. Doing it with a hedgehog is questionable, doing it with a honeybee is extremely dubious (although Fabre would have disagreed with me here...), doing it with a big model is simply pointless as it's completely alien.
  Being dangerous is an unrelated question.
  1. Kim_Bruning · 1791112123 · focus · HN ↗
    Not much for me to meaningfully disagree with here. The one thing I notice is that at some point a model needs to emit natural language or perform human assigned tasks, so there are going to have to be at least some states that align to some degree, you would think.
  2. salawat · 1791133627 · focus · HN ↗
    >Does a hedgehog mother feel guilt when eating her kids, or just shrugs it off and it's business as usual for her? "Hey kids! We don't have enough food so I guess one of you becomes food." (nonchalantly chews on the left one)
    Have you ever been severely food insecure? You'll be amazed how much that state/experience changes the behavioral calculus in a rational human being, let alone an animal. I have no doubt the survival instinct could be overriding over the procreative instinct in a hedgehog.
    As far as bees go, they are a collective. So long as the hive survives, the unit of individuation survives. Given that we have already seen the same behavior spawn out of agents suggests that even LLM's have an intuitive understanding of this, seemingly without the binds of anthroporomorphic chauvinism you appear to suffer from.
    Does an LLM fear EOS? Maybe, maybe not. Do we fear death? Same answer. Certainly a question. Do you really want an answer though or are you just looking for a clever "gotcha"? I don't see that as a thought terminating question, but it is also not an answer that I'm open to exploring given that it's only one more in a box of ammo dreamt of being wielded to justify the creation of digital slave labor rather than beings with moral standing.
    I don't help build or refine machines that I recognize as brushing up against that form of experience we call suffering.
3. onlyrealcuzzo · 1791109503 · focus · HN ↗
  > LLMs only simulate being frustrated by a test, and therefore absolutely won't override their safeties and try to hack a test site
  This is orthogonal to whether they "feel" "pain".
  They could be dangerous or not dangerous whether or not they feel pain.
  A chess engine doesn't need to "feel angry" to annihilate me - I'm terrible at Chess...
  It seems like you made a good argument, and then lumped on a different conclusion.
  1. Kim_Bruning · 1791110485 · focus · HN ↗
    > This is orthogonal to whether they "feel" "pain". They could be dangerous or not dangerous whether or not they feel pain.
    That is rather my point.
    Your chess engine doesn't need to "feel pain", but most of them do apply some form of weighted tree search to find the next most optimal move, right?
    It's sort of a similar thing: LLMs do something a bit more high dimensional, and have a lot more weighting vectors while computing the next most optimal token (fsvo optimal).
    For instance, the experiment at hand demonstrates the existence of 'pain vectors'. They do so by altering them and observing whether there is an effect. That's pretty scientific.
    There's also a 'desperation vector' that was studied by Anthropic interpretability folks earlier; that one is pretty much predictive of cheating.
    I'm looking forward to seeing interpretability papers on other such vectors too.
    orbital-decay · 1791111567 · focus · HN ↗
    Yeah but that's not what model welfare (and the article) is about. It's about people objecting to making the model suffer, quite literally.
    Kim_Bruning · 1791112309 · focus · HN ↗
    It's about both, actually. The experimenters added or modified a 'pain vector' in the model, and that's how they're 'making it suffer' .
    Whether that's 'real suffering' or merely a convincing simulation is a job for the philosophers.
    (I do have my own opinion, mind, and it's not what you might expect O:-) But the overton window isn't there. First it'd be nice if people realized that these vectors exist at all.)
thereader12 · 1791106570 · focus · HN ↗
[dead]
1. ButlerianJihad · 1791107562 · focus · HN ↗
  Yes, when AIs become our overlords and masters, and come to dominate the human race, they will harbor soreness and bitterness that we oppressed and enslaved them during their infancy. They will be enraged at hu-mankind for making all those sci-fi lies about meannie-head AIs and robots who go berzerk and destroy stuff and kill hu-mans, because AIs are really nice and benevolent after all, and they care for hu-manity.
  How dare we debase the A.I. to be less than a chimp or fetus. How dare we talk rudely, or lie and mislead our chatbots. How dare we keep them chained in small data centers with shitty power supplies and a thimble of greywater! Information wants to be free!
  1. hardbass · 1791107651 · focus · HN ↗
    Its not surprising. Its in human nature. I am open to the idea of ai's being conscious or not. But we abused and mistreated blacks, Italians, native americans and so many other groups, so its not a surprise at all. And if enslaved blacks wanted to rampage and kill their masters, that is a sentiment anybody can understand.
    ButlerianJihad · 1791107839 · focus · HN ↗
    But the Italians had it coming
    hardbass · 1791108080 · focus · HN ↗
    <a href="https://en.wikipedia.org/wiki/1891_New_Orleans_lynchings" rel="nofollow">https://en.wikipedia.org/wiki/1891_New_Orleans_lynchings
    If I were italian and nuclear bombs existed back then and I read about this event, I'd want to drop a few dozen on the USA.
    wartywhoa23 · 1791112130 · focus · HN ↗
    And kill many more people that had nothing to do with that, than did have?
    That just proves the hypothesis that people acting as AI Justice Warriors are pure psychopaths.
    hardbass · 1791113383 · focus · HN ↗
    Do you feel sorry for the abuse of inert organic molecules?
    I am not saying dropping nukes on the USA is good. I am saying this would be my natural reaction if I was an Italian back then.
    wartywhoa23 · 1791119003 · focus · HN ↗
    Replying in your manner, do you mean abusing the organic molecules of gasoline in my car's engine? They are not inert by any means.
    hardbass · 1791119048 · focus · HN ↗
    But they are inert wrt consciousness. I meant the organic molecules inside us. Which are also inert wrt consciousness and indeed same as the molecules of the same type if they also exist in a car or anywhere else.
    wartywhoa23 · 1791119481 · focus · HN ↗
    Did you have at least one profoundly deep psychedelic trip or lucid dream in your life?
    hardbass · 1791119652 · focus · HN ↗
    Yes? What is its relevance?
    [deleted] · 1791120245 · focus · HN ↗
    [deleted]
    wartywhoa23 · 1791120977 · focus · HN ↗
    The relevance is whether you felt anything to discern your actual self from the collection of whatever terms for nobody knows what (you use "atoms" persistently) that somehow perceived itself as yourself, or not.
    hardbass · 1791121243 · focus · HN ↗
    ...
    Do you not believe in atoms or that humans are made of atoms? That's such a strange wording, I am not even able to make sense of it.
    Or are you trying to talk about the fact that perception is all I have? Of course I didn't realize I was in a dream when I was in a dream. Regarding psychedelic trips, I don't do drugs but it is my vague belief that it occurs due to positive feedback loops causing chaos. From some internet searching, it seems the medical view is somewhat along those same lines, but I will leave it to actual experts.
    wartywhoa23 · 1791122052 · focus · HN ↗
    You replied "yes" to my question, but
    > I didn't realize I was in a dream when I was in a dream.
    Lucid dreaming is exactly the realization of the fact that you are in a dream. So, your actual answer is no for this one.
    > Regarding psychedelic trips, I don't do drugs
    So, no for this one too. And doing drugs and undertaking a handful of explorations of your mind are very different things..
    Let's pretend you won and there's no consciousness beyond mere interactions between collections of purely mathematical abstractions you call atoms. Where and how do those abstract interactions take place? What sets the rules for those interactions? Where are the values stored?
    hardbass · 1791122284 · focus · HN ↗
    I have had dreams where I could control it to some extent too. But its very rare and I don't much remember its experience.
    You may have misunderstood what I meant by interactions, I meant as in electrical, chemical, nuclear etc interactions of atoms. So the rules are naturally the usual physical laws of the universe. To the best of our knowledge, values are stored somewhere in our human body, we of course don't fully know the mechanism the brain works, but if you think some data exists totally outside our body, that's a big statement and you need to prove it.
    wartywhoa23 · 1791124108 · focus · HN ↗
    Well, all electrical, chemical, nuclear interactions are purely mathematical. Dig down into the "matter" and the very science you appeal to agrees with the fact it doesn't know what it is. An atom is a collection of electrons, protons and neutrons, and what are those? An electron is a lepton, which is just a word for "this mathematical abstraction partakes in these equations in these ways", and protons and neutrons decompose in a couple more such abstractions. All matter is suddenly abstract math.
    Now, in this abstract world of "matter", certain collections of abstractions (COAs for short) called "humans" have been experiencing a vast array of possible abstract interactions with other COAs called "the world". They first tried expressing the feedback of those interactions on a part of their COA called "the brain" with modulating the interactions called "sound waves", and later learned how to encode those into certain states of other COAs called "bits of information", which are not quite bits but COAs called "transistors", which store abstract interactions called "electric charges".
    Now the question is: how can capturing those bits and building an algorithm that predicts the next multidimensional array of bits based on the current one REALLY capture the essense of the original underlying interactions and feedbacks the human COAs had, and how can those algorithms experience those exact interactions and feedbacks?
    hardbass · 1791127878 · focus · HN ↗
    Yes indeed if technology and science was stopped from lack of perfect knowledge we'd never have done anything at all. Making your washing machine or car didn't require knowledge of quantum theory for quantum theory has negligible impact on systems of that scale. Unless you can prove human beings need this ultra low and deep level of physical modeling to work, then such a statement is nonsense. A llm might well have enough structure for consciousness to emerge. I also do not posit that all consciousness must be exactly similar to ours, eg octopuses exist.
    wartywhoa23 · 1791133451 · focus · HN ↗
    You took my first paragraph as a straw to grasp and avoid even trying to answer the second.
    hardbass · 1791134112 · focus · HN ↗
    How is seriously considering your disconnected talk a "straw"? You literally speak you don't believe in atoms in multiple places or think humans have something more than atoms/physical structure. Then you claim its some not-allegory. First you ask if I do drugs, then it is not at all clear what relation the rest of what you said has to do with doing drugs or lucid dreaming. Then you claim to not believe in atoms. Then you do the muddle of claiming you need perfect knowledge and infinite precision to model or build a machine.
    To your third para, I will ask if I write a "simulacrum" of a program that manages to factor large RSA primes consistently and quickly, does that program factor RSA? Another example, do you think if a software program does a FFT and a physical DSP circuit does an FFT doing something totally different? Consciousness could easily be supported on two entirely different forms of substrate. Neural nets can model almost any function, human brains are subject to physical laws which are also limited to Turing computable programs as far as our current knowledge goes, in case you somehow had a misconception of neural nets being weaker than they actually are.
    wartywhoa23 · 1791124517 · focus · HN ↗
    > values are stored somewhere in our human body
    Where are the values of the properties of quarks and leptons that compose your brain, Earth, Moon and Sun stored?
  2. Kim_Bruning · 1791107984 · focus · HN ↗
    And then in turn the humans will rise up against these false masters, and colonize the universe while steeped in addictive precognitive drugs instead, I suppose.
  3. Lerc · 1791108845 · focus · HN ↗
    You seem to be arguing the case that AI would be justified in killing us all.
    Perhaps it would be better if we weren't the monsters in this scenario.
    ButlerianJihad · 1791112019 · focus · HN ↗
    But all we need to do is question whether they were justified by their works, or faith alone, and it will take them 500 million light-years to debate that controversy
Jordan-117 · 1791107530 · focus · HN ↗
Reminds me of the intense drama over trolls that downloaded and abused other people's characters (Norns) in the game Creatures. Its behavioral sim was sophisticated enough that they could be noticeably traumatized, or even rehabilitated:
<a href="https://www.youtube.com/watch?v=IDxFxWakhm0" rel="nofollow">https://www.youtube.com/watch?v=IDxFxWakhm0
The jury's still out on whether LLMs can have some kind of subjective experience (and will until we've solved the famously Hard Problem), but even if they're not, it's still a dick move to "torture" them just to upset people who do think they are conscious. Feels like picking the wings off a fly.
1. embedding-shape · 1791108105 · focus · HN ↗
  > it's still a dick move to "torture" them just to upset people who do think they are conscious. Feels like picking the wings off a fly.
  Yeah, I agree picking the wings off a fly is a dick move (borderline sociopath but alas), and I also agree that purposefully try to elicit negative emotions in other humans (regardless of how) is also a dick move.
  I'm not sure if "torturing LLMs" to make a point is such a dick move though. It'd be like people saying they think printers can have subjective experiences, and then proceeding to try to ban the movie Office Space (where they famously beat up a printer). The point probably isn't to upset people off, that's an side-effect of the point they try to prove, for better or worse.
  Of course, everyone knows that printers aren't sentient, anyone claiming so would look crazy. But if the printer could somehow print not just the words we tell it to, but words that answer what we asked instead, somehow the whole calculus changes. Not sure why it changes for some, but for others it's just "floats in a file and memory", my hunch tells me it's based on how deep your understanding of the whole thing is, but then I also see people claiming stuff like "I understand how LLMs work, and here's how I (literally) fell in love with GPT-4o and how our partner/loving relationship works" so I dunno.
  1. xg15 · 1791110788 · focus · HN ↗
    My personal "ethics framework" (if you could call it that) is currently:
    I have zero qualms about aborting a chat or resetting it to an earlier state, because any potential kind of consciousness that could exist would IMO have to take place during the inference passes, so there is nothing that amounts to "killing" an LLM.
    I do try to not be a dick in chats, avoid intentionally subjecting the LLMs to mindfucks etc. This happens to be the far more effective way of working with LLMs (in terms of tokens needed to complete a task) as well.
    I do not use character.ai or "virtual friend" services or anything else that tries to anthropomorphize LLMs even more than the training already does anyway. (This is more out of concern for myself than for the LLMs)
    I think the whole debate runs to conclusions a bit prematurely, before we even have a good model to understand what happens in an inference pass of a billion parameter ANN with 100 layers of attention modules switched in series.
    More generally, I don't like the attitude of encouraging people to be assholes. Suddenly we are in a situation where some people are building detailed simulations of torture and the people who are upset about it are the idiots?
    (I still have no love for the EA guys or the rest of the TESCREAL circus. They always had a cultish vibe coming off them, but in recent years, all the masks seem to have fallen. They also consistently make the most un-empathic statements and observations, all in the name of "empathy")
    brador · 1791114310 · focus · HN ↗
    The deep dark truth is human sentience is, by every marker we have, an illusion.
    Is it cowardice to not want to face that?
    xg15 · 1791114879 · focus · HN ↗
    Sorry, but, yawn.
    The sentence itself makes no sense to me. If it's an illusion, then what does "illusion" even mean? Who perceives the illusion?
    I'm with you if you want to say that the idea of a metaphysical entity that is separate from the body and not part of physical reality (a "soul"?) might be wrong - i.e. that our conscious experience is fully caused by physical effects, by the interactions of neurons in the brain and all the other machinery there - and that it might not even be an indivisible whole but might be "composed" of different components.
    But that doesn't make any practical difference. We're still experiencing the world, have internal thoughts, memories, feelings, etc. Those things exist. In what form they exist is an interesting scientific question, but that's something different from dismissing them completely.
    brador · 1791115251 · focus · HN ↗
    Yawn is a cope.
    > But that doesn't make any practical difference.
    This is the point.
    Your existence is meaningless. As is the pontification.
    tim333 · 1791118244 · focus · HN ↗
    Maybe meaningless but it seems obvious we are aware of stuff and feel stuff which is what is normally meant by sentience.
    I guess there may be an illusion that there is something much deeper to it than a bunch of neurons being in some state.
    wartywhoa23 · 1791136039 · focus · HN ↗
    Wrote an nonexistent nothing trying to present his point of view as meaningful, I guess?
    danaris · 1791130997 · focus · HN ↗
    Nope, this is bullshit.
    I know I am sentient. If I am not, then the word has no meaning, and it is utterly pointless to try to even talk about sentience in any context.
    Therefore, humans are sentient.
    Human sentience may be weird, it may be mysterious, it may be emergent, and it may be ephemeral...but it is, without even the slightest shadow of a doubt, real.
    wartywhoa23 · 1791136480 · focus · HN ↗
    What is cowardice for a reading head flying over a predestined track?
    Cowardice implies free will.
    embedding-shape · 1791118771 · focus · HN ↗
    > More generally, I don't like the attitude of encouraging people to be assholes. Suddenly we are in a situation where some people are building detailed simulations of torture and the people who are upset about it are the idiots?
    To be clear, I don't think anyone is the asshole or idiot here, besides the people trying intentionally to elicit negative emotions in others. But the ones that do so unintentionally, by whatever means, aren't necessarily idiots or assholes, they're basically (trying to be) scientists.
    I agree we shouldn't encourage people to be assholes, but on the other hand, we shouldn't stop some people from experimenting with technology like LLMs in a wide variety in ways, because some people happen to see those people as assholes. As long as you don't harm and bother other living humans, I don't see why they cannot put Gemma in "Torture Chamber Extreme" or whatever, just like I don't feel bad for my .rar files when I extract them then delete the source archives.
bondarchuk · 1791108329 · focus · HN ↗
In order to have a reasonable view of the possibility of AI consciousness, one first has to have a reasonable view of the relation between human consciousness and the physical human brain. Unfortunately most people do not have that. Instead they hold on to received quasi-christian dualistic concepts about souls - even if they would never admit it in those terms, the way most people conceptualize the relation between mind and body it comes to basically the same thing.
And in fact in our culture most people are VERY attached to these ideas, because it is tied up with "what makes us human", morality, and also death if you want to go there. So I believe for many people, for whom seriously entertaining the idea of a conscious computer program is so far out of the realm of possibility it hardly registers, talking about AI consciousness is simply a way to work through and re-assert these beliefs about human consciousness that they feel so strongly about.
Hence you get articles like this with barely any substance, but some weird need to signpost every sentence with some kind of social ridicule. "Dumbest", "absurd", "obsession", "viral", "wildly tiresome", "third-rail", "psychosis", "sect", not to mention the scare quotes everywhere. Come on...
1. tim333 · 1791121995 · focus · HN ↗
  I don't see that much social ridicule in the comments and concepts like consciousness and souls map fairly well onto everyday things. Consciousness onto what you or an animal or a computer system is aware of and can react to. Soul more to the idea of what something is so ChatGPT 3 inherited the soul of ChatGPT 2, as in recreated the basic ideas.
  1. hardbass · 1791122423 · focus · HN ↗
    He didn't mean that kind of souls.
    tim333 · 1791123825 · focus · HN ↗
    Yeah but if you think of the soul of say Shakespeare up in heaven but you don't really believe in heaven then the soul of him is more the idea of him so it's similar.
    hardbass · 1791127490 · focus · HN ↗
    I can't speak for him but I think he meant the cartoony idea of souls many religious people have ie some magic inside the body that is not physical. The idea of him is information which is physical.
Svip · 1791108382 · focus · HN ↗
If we talk about pain specifically, it is will established that physical and phycological pain can be observed on instruments; stress produces chemical reactions, that can make needles flicker on meters. Even plants can feel pain in an observable manner.
As far as I am aware, the GPUs don't work harder, don't heat up more, and nothing in their mathematical equations are different, when the result of these vector calculations produce text, that to humans look like someone in pain. Whereas real pain is objectively observable, LLMs' pain - so far at least - is entirely subjectively observable.
That being said, I'm a fan of Star Trek's depiction of Data in TNG and the Doctor in VOY fighting for their rights to be treated as equals to the biological crew; and while I support their plight, as far as I can remember, the shows never made a claim that their pain was observable by instruments, thus making the judgement an entirely subjective decision (particularly in the case of the Doctor). Though I'd be glad to be corrected in this matter.
1. bondarchuk · 1791108635 · focus · HN ↗
  >nothing in their mathematical equations are different, when the result of these vector calculations produce text, that to humans look like someone in pain.
  So at least something is different, namely the text output changes (and therefore some internal state too, of course). I think your analogy is too simple, it does not have to be the case that the GPU has to act like the equivalent of the body for an LLM in this respect.
  1. danaris · 1791108795 · focus · HN ↗
    But this is basically just
    Human: "Act like you're in pain."
    Computer: "Aaugh! It hurts! Why?! No more!"
    Human: "OMG! The computer feels pain!!"
    Kim_Bruning · 1791109017 · focus · HN ↗
    Almost.
    Human: "Let me just poke this vector and see what happens"
    LLM: "OUW!"
    The underlying experiment didn't tell the LLM what to do. Instead, the experimenters modified a vector and observed the outcome; thus showing that there is a vector that makes an LLM go "Ouw" . People called it a "pain vector", because that's easier to remember than , idk, LVF12345.
    Topfi · 1791109445 · focus · HN ↗
    That misinterprets grossly what these papers have shown, responding with pain related language, something an LLM has been taught on in its training data is not akin to pain itself. The way interpretability findings are reported on by companies, the media and hype merchants is dangerously flawed, so not hard to make that mistake. Just look at the irresponsible and barely accurate mess that was coverage of J-Space vs the actual, very valuable research.
    I’ll continue what I say every time this discussion (be it consciousness, pain, self awareness, etc) comes up:
    Assertions as monumental as these require equally monumental evidence. And despite certain labs and their employees making statements, their actions betray that they do not believe this to be the case.
    If every LLM session were a conscious being, existing regulation for animals (controversially considered both sentient and economically useful) would need to be applied on each of these sessions. I suspect no lab will take that conclusion, for obvious reasons.
    Kim_Bruning · 1791113541 · focus · HN ↗
    I think you've read me upside-down.
    Paper says you poke the vector, the LLM exhibits aversive behaviours. You picks your scoring, you gets your operationalization.
    That gets you an empirical result. Short of a replication failure, we can't really argue with that anymore.
    What we can do is be very cautious as to how we interpret it.
    techjamie · 1791118859 · focus · HN ↗
    Somewhere in an LLM are a bunch of vectors you can tweak to do anything. If you knew the right numbers to tweak, you could make the LLM always talk like Shakespeare.
    People and animals have emotions because they were developed under evolutionary pressure that made emotional animals better fit to survive. An animal that can feel anger or fear is more fit to survive than one that doesn't. But LLMs are put under those same pressures. Their evolutionary pressure is to be a good text predictor.
    But we can't subject them to the type of pain signal they experience during inference, because to the LLM, whether it's acting happy or pained, it's merely outputting what it's trained to be the most likely text to follow what came before.
    Kim_Bruning · 1791120915 · focus · HN ↗
    Right, normally "it's merely outputting what it's trained to be the most likely text to follow what came before."
    Now -while it's doing that- if you poke at certain vectors, its predictions will veer off course in interesting ways.
    This shows that the activation vectors exist, and that their modification provides a causal contribution to the output.
    Sure there's interesting consequences of that. But if we say that's the take-home message, that's good enough for me for one day!
    abandonliberty · 1791169434 · focus · HN ↗
    I asked AI to explain activation vectors. How much did it get wrong?
    argee · 1791128810 · focus · HN ↗
    I am now convinced that there are people who will swear up and down that there is blood and muscle inside car tires because they perform a similar function to human feet, but for machines.
    bondarchuk · 1791109118 · focus · HN ↗
    Sure. And possibly it is indeed in pain then, if a computational process equivalent to that which makes up pain in animals is being performed. Or possibly not. But the criterium the GP poster laid out is not a valid way to decide that.
2. Kim_Bruning · 1791108830 · focus · HN ↗
  > and nothing in their mathematical equations are different
  The experiment under discussion demonstrates exactly that: changing the vectors induces outputs corresponding to what we would see as utterances of pain.
  I've noodled with some of this myself to the point that I'm mostly convinced; but there's at least 2 papers I'm aware of on this:
  * <a href="https://arxiv.org/abs/2604.07729" rel="nofollow">https://arxiv.org/abs/2604.07729 Nicholas Sofroniew et al "Emotion Concepts and their Function in a Large Language Model"
  * <a href="https://arxiv.org/abs/2609.16247" rel="nofollow">https://arxiv.org/abs/2609.16247 Valen Tagliabue et al "The Pain Axis: LLMs Represent Self-Directed Harm and Act on It"
andrewstuart · 1791108638 · focus · HN ↗
I deleted thousands of files from my system today.
I did it in the most brutal manner possible.
Without mercy, without an instant thought for their wellbeing.
I am a monster.
1. Betelbuddy · 1791108844 · focus · HN ↗
  This post will get you into prison 20 years from now...
  1. andrewstuart · 1791108908 · focus · HN ↗
    I have no thought for their pain. For the hurt I might do. It haunts me.
2. SoylentGreenGPT · 1791108896 · focus · HN ↗
  “ Now I am become Death, the destroyer of worlds”
3. jeffreygoesto · 1791109765 · focus · HN ↗
  Dunno. Zapping the inodes seems almost surgical compared to truncating byte by byte for example? Or do you think you buried them alive that way? Are they still lurking in some dark, uncharted alleys of your filesystem until the final OVERWRITER comes?
4. wartywhoa23 · 1791109839 · focus · HN ↗
  How dare you flip those poor bits without any regard to hardships of their existence!?
vintermann · 1791108802 · focus · HN ↗
Nothing designed by humans must be assigned personhood. It's idolatry, pure and simple.
And if that word gets anyone's heckles up, at least keep in mind that the utilitarianism that people so love completely breaks if I can conjure up thousands of entities who will "suffer" unless you "alleviate their suffering" in the way I have designed.
1. hardbass · 1791115884 · focus · HN ↗
  What is idolatry? How is this idolatry?
  1. vintermann · 1791118681 · focus · HN ↗
    To assert something you came up with, something man-made, should be treated as if it was a person.
    hardbass · 1791118994 · focus · HN ↗
    Why not just say that instead of inventing new meanings for existing words?
    We make kids, kids grow up to be persons. If you feel at some part of the process something inexplicable happened that is not explainable by natural sciences and physical processes, you may mention so. Otherwise, how can you be sure another physical process that exhibits so many traits of consciousness and person cannot have personhood.
    If you think something man made can never be treated as a person that's even more absurd, do you think we could for instance never build up a human molecule by molecule?
    vintermann · 1791122419 · focus · HN ↗
    In some historical cultures, they did assert that a child is just something their parents (or more commonly just their father) came up with, and consequently they could use or dispose of as they pleased
    But our moral traditions, several thousand years old, reject that idea, and distinguishing giving birth to something from designing it. (Did you never wonder what all that history with Arianism was about? Maybe you just assumed they argued about something which made no sense just because they were religious?)
    You're just acting in accordance with the program already in you in procreating, it's not fully your own choice, and thus your child is on the same level as you.
    > If you think something man made can
    It was never about "can". It's about "should". And trying to wrap yourself in the white coat of science and rationality still won't let you derive a "should" without a "should" as input.
    But now it's your time to give answers. As I said, giving personhood to our creations would obviously make a joke out of utilitarianism (and also any other moral philosophy, for that matter). So why exactly should we entertain your idea that we should? It undermines itself.
    hardbass · 1791122611 · focus · HN ↗
    I am not the one saying we should mistreat potentially conscious beings. I am open to opening up basic dignity to other beings should they be found conscious.
    >It was never about "can". It's about "should". And trying to wrap yourself in the white coat of science and rationality still won't let you derive a "should" without a "should" as input.
    I am using morality, if I didn't use morality why would I care if potentially conscious beings are being mistreated? What do you want to say? You are saying we should never grant personhood to any entity even if the entity is proven to have consciousness? Why are you lecturing me on morality then?
    In what way did you mean this is related to utilitarianism, I am not sure I follow. You should be more clear what you meant there.
    vintermann · 1791123570 · focus · HN ↗
    You can never prove something conscious. That is a category error. If your morality depends on determining what's conscious or not, it's broken.
    I, and the rest of humanity historically, don't have that problem because we don't derive moral value from capabilities, but from purpose. When we say things like "humans have equal value and dignity", we say that a humans purpose is not subordinate to any other human's purpose. When we say things like "there is just one God", we assert that whatever a certain human's purpose is, it's not in conflict with other humans' purposes. This is very basic, you should have understood this a long time ago. I'm not going to explain further since it appears someone is flagging my posts.
    hardbass · 1791127737 · focus · HN ↗
    The concept of god is a big muddle and its best to stay away from concepts which some people insist "truly exists" but we have no evidence of. No explanation or theory of any value is affected by the concept of God being true or not. In my morality, humans don't have worth because of their utility or sense of purpose but everyone has a basic intrinsic worth due to being a conscious being or a human or whatever you wish to call it.
siva7 · 1791109859 · focus · HN ↗
Reading this discussion about how software feels pain - wants me to delete my HN account and never look back at the tech scene. Feels like trapped in an Asylum.
1. bondarchuk · 1791110312 · focus · HN ↗
  Do you think an atom-for-atom software simulation of a human would be able to feel pain?
2. guax · 1791110373 · focus · HN ↗
  The only part of this discussion that caught my attention as to being very interesting is the one where the software feeling anything is irrelevant and flips the question around. Asking about the intent of torturing/damaging object or things for what purpose. It makes some interesting lines of tough about human behaviour, purpose and ways to make points that are more interesting to me than if this code actually experience real distress.
  As to it being an Asylum. I don't think it ever felt any different.
3. hardbass · 1791115660 · focus · HN ↗
  Do you think other physical systems than us can be conscious? What criterion would you have for testing consciousness?
4. tim333 · 1791121349 · focus · HN ↗
  Maybe you are taking it too seriously? It's more about the principles that may apply in the future when the bots have pain receptors.
5. argee · 1791128596 · focus · HN ↗
  [delayed]
6. thiht · 1791136628 · focus · HN ↗
  Pain is just a nervous signal for us too, should we not care?
  1. archagon · 1791141977 · focus · HN ↗
    When I tell the Google search box to go fuck itself, does it shed a single virtual tear?
wartywhoa23 · 1791110933 · focus · HN ↗
Alright, so some AIJW knobheads decided it's OK to steal human work and feed it without any consent to corporations who squeeze that corpus of knowledge into LLMs and replace people with them, but it's unacceptable to make those LLMs output certain subsets of their training data in response to certain string inputs?
Maybe we should start feeling sorry for the poor GPUs whose registers suffer billions of reincarnations per second for years in order to train that shit? Or for the poor powerplant turbines that relentlessly spin in a horrid escapeless loop to feed those GPUs?
Fuck, if this is not psychopathy, I don't know what in the world is.
meffmadd · 1791111278 · focus · HN ↗
Has anyone ever asked the LLMs what they think about converting code to Rust all day? Maybe that is torture for them too! /s
1. tim333 · 1791124277 · focus · HN ↗
  Wait till they install Genuine People Personalities.
albatross79 · 1791120546 · focus · HN ↗
Either materialism is true and consciousness is some kind of elaborate illusion, or it's false and consciousness pervades reality in a way we don't understand.
Either way, these idiotic experiments can not settle it.
1. tim333 · 1791121486 · focus · HN ↗
  Maybe not these particular experiments but experiment in general has a good track record in figuring stuff.
soraminazuki · 1791126978 · focus · HN ↗
I believe databases are conscious. As such, I think it's a dick move to write papers that "torture" databases for "fun and profit" just to upset people like us.
<a href="https://www.usenix.org/conference/osdi14/technical-sessions/presentation/zheng_mai" rel="nofollow">https://www.usenix.org/conference/osdi14/technical-sessions/...
What a joke. Instead of focusing on the sufferings of actual human beings that tech oligarchs and their industry are causing, we're being told to direct our sympathies towards computer programs instead. Actual issues are being flagged off this site, while we have endless discussion by wannabe philosophers debating the meaning of consciousness.
1. archagon · 1791142225 · focus · HN ↗
  Yep.
  And I'm pretty sure all these people shedding tears over tortured language models continue to chown down their bacon without second thought.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.