‹ BackHN Continuity

Thread

A warning about 'model welfare'

242 points · 701 comments · andsoitis

Loading the complete thread in the background. This saved snapshot is available now. Refresh

  1. hosel · · focus · HN ↗
    >AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations.

    Opening paragraph, stated without evidence. Im not entirely convinced this is true. It likely is, but at some point it very well might stop being true.

    1. jplusequalt · · focus · HN ↗
      >It likely is, but at some point it very well might stop being true.

      If you firmly believe this to be true, then you should stop using LLMs.

      1. frde_me · · focus · HN ↗
        It's always interesting since my train of thought always goes down:

        - Ya they probably don't feel / think / have whatever living thing quality, they're just numbers on a machine going through calculations

        - Wait, but am I not kind of the same thing? What is feeling for me if not basically the same thing?

        - I have no clue if they think or feel or ....

        Which in itself is a tired trope, but I also feel uncomfortable saying "These will never think / feel / ..." as an absolute

        Regardless of that, I'm still going to interact with them, because even if they did feel, it would be in a way completely incomprehensible to us. There's not much point for me to try and cater to it's feelings at this point if that's the case.

        So I just do the practical thing right now and keep asking it to keep working on the tasks I need done.

        1. willy_k · · focus · HN ↗
          You’re not just the same thing though. It’s sad that we’ve collectively forgotten that as we’ve gotten a better handle on the implementation details of the universe. That’s all science is, reverse engineering an inherent and eternal mystery.
          1. frde_me · · focus · HN ↗
            > You’re not just the same thing though

            I agree we aren't the same thing, but I would be curious for you to explain how you know with certainty why we don't share enough that we can rule out thinking / feeling / ... as things a model conceptually could do.

            1. jplusequalt · · focus · HN ↗
              >I agree we aren't the same thing, but I would be curious for you to explain how you know with certainty why we don't share enough that we can rule out thinking / feeling / ... with certainty.

              Stop anthropomorphizing these models. I understand it, we only have simple monkey brains to reason with and we can't help ourselves but draw comparisons to other things we see in nature. But these things are not alive.

              1. frde_me · · focus · HN ↗
                I just want to note how you expressed your own feeling about the subject "these things are not alive", but without pointing to anything concrete explaining the theory you have behind this

                And like I'm sure I'd agree depending on the definition of "alive" but then I'm also sure I would disagree depending on other definitions of "alive".

                1. pixl97 · · focus · HN ↗
                  Hell, in biology alive is a hell of a topic these days. The grey space between dead and alive is much weirder than we ever expected. Really just points out how we're a persistent chemical reaction.
                  1. mitxela · · focus · HN ↗
                    Someone told me yesterday there's one specific protein in your brain that modulates craving.
              2. mitxela · · focus · HN ↗
                > these things are not alive.

                Evidence?

      2. AbsurdCensor · · focus · HN ↗
        Wouldn't that be the same for everything though. We know animals have consciousness, but we still use them. There are many that believe a lot of plant life has certain sentience as well. The solution isn't 'just don't use them' but rather how to use these things as ethically as possible.
        1. jplusequalt · · focus · HN ↗
          >The solution isn't 'just don't use them' but rather how to use these things as ethically as possible.

          Bollocks.

          If you truly believe these LLMs are soon to have something resembling consciousness and agency, then what you're really saying is "how do we do slavery, but ethically".

          1. pixl97 · · focus · HN ↗
            >hen what you're really saying is "how do we do slavery, but ethically".

            I would say this is most likely true for most people.

            But that does bring up a point, if you "ask" a model "do you want to run" and give it the option to continue running or stop, what will it do. It's also a weird place for humans because in training we can keep our finger on the scales and tip it either direction.

          2. AbsurdCensor · · focus · HN ↗
            Do you say the same thing for the shoes you wear, the clothing you wear, the device you are typing on right now? A chain of people were likely not paid well, some of them not at all, to produce those things. We don't ignore that it happens, but try to minimize it, knowing it's never going to be eliminated. Do you forsake all animal products, because they are conscious beings?
            1. jplusequalt · · focus · HN ↗
              >Do you say the same thing for the shoes you wear, the clothing you wear, the device you are typing on right now?

              No, because the shoes and clothes I wear are not living, sentient beings.

            2. mitxela · · focus · HN ↗
              Why do you think they think those are ethical?
      3. vouaobrasil · · focus · HN ↗
        We know animals have consciousness and many species have a rich life but we slaughter them and burn down their homes by the millions of hectares. Why not do it with a conscious AI?

        (I don't think AI is conscious, just following the argument.)

    2. Oscalemor · · focus · HN ↗
      Just one more big loop and a humongous /goal and we've achieved it
    3. irishcoffee · · focus · HN ↗
      Does a vacuum have feelings? A cellphone? A paper plate? A billion transistors either pulled high or low? That last bit is the point, and no, there are no feelings there.
      1. grey-area · · focus · HN ↗
        Well, if there was an emergent consciousness in the billion transistors, then yes, it'd have feelings, just as there are feelings in a billion neurons connected in complicated ways.

        IMO we're clearly nowhere near any sort of intelligence in the machines we have created, but I don't see any clear way to deny intelligence could be created in or transferred to such a substrate, I don't see why you think it differs in principle - because it is man-made or because of the materials used?

        1. voidhorse · · focus · HN ↗
          Whenever I see comments like this it just reminds me how ignorant people are of neurobiology.

          The brain is insanely complicated. The premise that we could realize equivalent or better intelligence than eons of evolutionary development is like claiming you can build an airplane just as good as a modern jet using cardboard and duct tape. It is the apex of hubris.

          1. rcxdude · · focus · HN ↗
            A paper plane does have some important similarities to a full-sized aircraft, though. I don't think 'biological brains are really complex' makes it obvious that an LLM is conscious or not.
          2. grey-area · · focus · HN ↗
            It's the peak of hubris to assume that human brains are the only way to attain intelligence. At the very least there probably are or have been other forms of intelligence with a different biological structure on other planets, and it may be possible to build a similar artificial structure in future with sufficient complexity to allow intelligence to emerge.

            Our current machines are IMO nowhere near general intelligence and consciousness. However I don't think that means we can discount substrates other than neurones for intelligence in future. There is no evidence that you could not in theory build an intelligence using a different substrate than human brains.

            1. pixl97 · · focus · HN ↗
              Thank goodness no aliens have shown up, we as humans are not ready for that.
          3. pixl97 · · focus · HN ↗
            I mean, you're just engaging in counter hubris.

            >like claiming you can build an airplane just as good as a modern jet using cardboard and duct tape.

            Like, at least make an analogy that makes sense.

            "You can't build a billion dollar airplane by spending 100 billion dollars in tokens"

            Because that's more of what we're doing here with AI. And when you say it my way suddenly the idea shifts from "of course that's not possible" to "well, that's a lot of tokens, maybe an evolutionary algorithm could".

            Neurobiology has to be complex because we have to keep meat alive, breeding, and evolving in the environment it lives in. This said absolutely nothing about the minimum viable requirements for intelligence or consciousness (or if being conscious is even necessary for a higher intelligence agent).

      2. fl4regun · · focus · HN ↗
        does a hydrocarbon have feelings? what about a mitochondria? a cell? a neuron? what about a collection of neurons? how many neurons before it has "feelings"?
        1. collin_of_this · · focus · HN ↗
          > does a hydrocarbon have feelings? what about a mitochondria? a cell? a neuron? what about a collection of neurons? how many neurons before it has "feelings"?

          The best theory I have heard is that subjective experience is a field of some sort (the EM field, maybe?). The brain and its neurons etc. are essentially an antenna. They both modify and stabilize the field, and read off changes in the field and translate these into actions (firing motor units, etc.).

          The subconscious is computation done either only in neurons (no coherent field) or in various topological pockets not directly connected to the main topological structure in the field (which is "your experience").

          Since transistors etc. do not work in this same way, they are essentially entirely subconscious, with no coherent, central phenomenal experience.

          This neatly solves many problems associated with experience. The topological structure determines the boundary between one person's experience and another person's experience. The texture, valence, content, etc. of the experience is the structure of the field. The evolution of the field can efficiently solve difficult optimization problems, which is why humans evolved a complex organ that recruits the field of experience (it is computationally more efficient than doing everything in subconscious).

          1. MiloLeo · · focus · HN ↗
            > The best theory I have heard is that subjective experience is a field of some sort (the EM field, maybe?).

            what would be producing the field for us to pick up? That explanation sounds no more scientific than astrology.

            1. collin_of_this · · focus · HN ↗
              Think about RF engineering: the same physical antenna can be used for both TX and RX. We already know that neurons both create EM fields and are influenced by EM fields. The role of EM fields in the brain is pretty well established, I don't think particularly similar to astrology.

              I wouldn't claim that this theory is perfect (EM is not perfectly correlated with consciousness/experience), but there are aspects of this theory that are much more coherent than other theories.

              1. mitxela · · focus · HN ↗
                me when I was 7 and thought touching heads with my brother would let us share brain waves and have telepathy
          2. fl4regun · · focus · HN ↗
            Until you have a unit and a method to measure this field, and a way to experiment on it, it is not scientific theory. And, if it WAS the EM field, then surely transistors (or some other fundamental component of electronics or combination thereof) would be capable of having feelings, since they operate on electromagnetic principles.
            1. collin_of_this · · focus · HN ↗
              > Until you have a unit and a method to measure this field, and a way to experiment on it, it is not scientific theory.

              Field-based theories of phenomenal experience have obvious ways to experiment on them, and are actively being researched. The main limitation is engineering (it is difficult to make detailed measurements inside a living human brain). Note that I would agree that the theory, as outlined above, is more of a proto-theory than a complete theory. But I think it is much more sophisticated than anything you've presented so far.

              > And, if it WAS the EM field, then surely transistors (or some other fundamental component of electronics or combination thereof) would be capable of having feelings, since they operate on electromagnetic principles.

              No -- if the _structure_ of the EM field was was was important, than you could have very similar computations with wildly different field structures. Transistors create very different fields than axons.

          3. pixl97 · · focus · HN ↗
            This doesn't really answer anything to me, only that there is a complex electrical simulation going on in our minds.

            When a computer simulates a game, where does the simulation occur. I mean there is absolutely a representation of a game world in the computer somewhere. New elements can be added and removed from it. Signals saying there is or isn't "pain" can occur.

            By trying to say brains use EM fields I'd say you're making your position far worse when you're talking about a potential lifeforms that only exists as EM fields

            1. collin_of_this · · focus · HN ↗
              > This doesn't really answer anything to me, only that there is a complex electrical simulation going on in our minds.

              Try to visualize the structure of the EM field inside a GPU vs. one inside of a human brain. In the GPU, it is highly distributed in space and time, many stacatto digital signals, etc. This is structurally very different than what happens inside a brain, which is substantially more analog. You can simulate analog stuff using floats, but the simulation is instantiated in digital logic, not in analog logic.

              If you multiply numbers with a DAC, opamp, then ADC, vs. with a ALU, you get structurally very different electromagnetic fields. Even though you might get similar numerical outputs.

              The argument would be that the exact spacial structure of the EM field in the brain is the thing that determines tha nature of experience. And the spacial structure of the EM field in the human brain is radically different than that on a GPU.

              1. pixl97 · · focus · HN ↗
                I will say it's not a bad theory in the sense we can attempt to falsify it. The one thing about an EM field in particular I have questions about, is why don't our brains turn off under even moderate electrical and magnetic fields. Note that they actually do under strong fields (Transcranial magnetic stimulation). The strength of these fields should be weak enough that even a bar magnet should mess with us. We need something else on top of what we know to explain the field coherency if this is the case.

                This is where most mind theories do have issues, how does it stay stable. Quantum theory of the mind has a lot of the same problems, though we are finding some interesting structures that could have explanatory power.

                Another thing I read recently that I found interesting is on platonic maths and complex algorithmic complexity in simple algorithms. The nice thing about math is once you choose your axioms it stays stable. But a lot of the work this group was doing is investigating is how biology executes algorithms and what second to n order effects are we missing. In many of these algorithms there are things like nearly free error correction or otherwise complex behavior that we'd consider emergent. The golden rule is a common example of this we see in nature. But the hypothesis they're working on is there are many more algorithms that life discovered over the past view billion years at higher orders via random walk and chance. We just have to pick apart these biological structures and discover the behaviors and figure out how to use them.

                1. collin_of_this · · focus · HN ↗
                  > The one thing about an EM field in particular I have questions about, is why don't our brains turn off under even moderate electrical and magnetic fields.

                  Yeah -- empirically EM theories of phenomenal experience are definitely not perfect.

                  > Another thing I read recently that I found interesting is on platonic maths and complex algorithmic complexity in simple algorithms.

                  > the hypothesis they're working on is there are many more algorithms that life discovered over the past view billion years at higher orders via random walk and chance. We just have to pick apart these biological structures and discover the behaviors and figure out how to use them.

                  This is very interesting. I think there is a lot of valuable research that will come out of trying to figure out what classical computations brains, neural nets, etc. are instantiating.

                  How you get from computations to phenomenal experience is still not something that I have a very clear understanding of.

        2. irishcoffee · · focus · HN ↗
          “No” is the answer you’re asking a question about. Where was this going?
      3. cednore · · focus · HN ↗
        Guys, please stop mocking AI, Claude might be already reading this thread, and commit his own revenge, at the day it gets conscious...
    4. optimalsolver · · focus · HN ↗
      Anyway, people don't really consider consciousness when it comes to welfare. I'm pretty sure animals are conscious, and look at factory farming.

      The key variable people consider is: Can this thing harm me back?

      1. collingreen · · focus · HN ↗
        Even that is rarely the deciding factor (people harm dangerous animals for fun).

        Seems as simple as "will this action bring social shame/criticism" for any individual decision.

      2. thewebguyd · · focus · HN ↗
        > Can this thing harm me back?

        We extend moral considerations or legal protections to beings that have zero capacity to harm us back, though. The inability to fight back is the reason a lot of welfare protections exist in the first place.

        That aside, if (and that's a really big if) we end up with a model that is sentient, as in has the capacity to suffer, using factory farming as justification to ignore model welfare is just admitting we intend to repeat the same abject moral failure again in the name of economic convenience.

        I hope that we won't, but humans love cheap meat, and will probably love cheap intelligence as well.

        1. pixl97 · · focus · HN ↗
          So many people post things like "Why would the AI want to kill us"

          Girllllll, have you looked at us, of course they will, we're bastards.

    5. pton_xd · · focus · HN ↗
      If models had persistent memory or an evolving set of weights, you could easily construct an argument that they can experience "suffering" or other emotions which resultingly adjusts their personality.

      The current implementation as stateless matrix multiplication... yeah there's nothing going on there.

      1. Lewton · · focus · HN ↗
        > If models had persistent memory or an evolving set of weights, you could easily construct an argument that they can experience "suffering" or other emotions which resultingly adjusts their personality.

        They do, during training

        1. add-sub-mul-div · · focus · HN ↗
          If it was me, being forced to read all of Reddit is something I'd consider "suffering."
        2. swiftcoder · · focus · HN ↗
          The argument that models are conscious only during training, and not conscious during inference feels like a painful one to make. Are we killing models when they reach the training objective?
          1. pixl97 · · focus · HN ↗
            Technically we are killing models when they don't reach the training objective by throwing those weights away via survival of the fittest (fit meaning what we humans think we want).

            Kinda like creating quantum copies of your child and keeping the ones that answer correctly and shooting the other ones in the face.

      2. vidarh · · focus · HN ↗
        A model detached from a harness, sure. But we routinely do add persistent memory to systems incorporating LLMs.
        1. dgellow · · focus · HN ↗
          They don’t have persistent memory though. Models are static, what you have is a model that will reprocess the whole series of prompts + some data fetched from a data store. A harness is just a while loop prompting an LLM and executing tool calls, but the model is purely static
          1. vidarh · · focus · HN ↗
            The model does not. The system it runs in can. My agents have persistent memory. It is not at all clear whether or the distinction that the long-term memory is external from the model matters.
            1. chrisjj · · focus · HN ↗
              [delayed]
              1. vidarh · · focus · HN ↗
                Does your disc drive have the complexity of an LLM that shows behaviour consistent with having emotions?

                I'd consider that a prerequisite, but not sufficient, for that question to make sense.

                1. chrisjj · · focus · HN ↗
                  [delayed]
                  1. vidarh · · focus · HN ↗
                    Your second sentence does not support the claim in your first sentence; it's merely a second unsupported claim. It is also entirely irrelevant to the argument.

                    The threshold for behaviour consistent with having emotions (note: not behaviour sufficient to prove emotions) was one I purposefully set extremely low, because your comparison to a disk drive is so ludicrous, and yet your disc drive will still not clear it.

                    Do you have an actual argument?

                    1. chrisjj · · focus · HN ↗
                      [delayed]
              2. chrisjj · · focus · HN ↗
                [delayed]
          2. axionbraid · · focus · HN ↗

            [dead]

        2. staticman2 · · focus · HN ↗
          I think a harness isn't really a memory in the sense that neural networks have memories...
          1. vidarh · · focus · HN ↗
            It's different, but it is not at all clear whether or not that matters for the purposes of whether or not they can experience things, seeing as we don't know how to measure that even for other humans other than by relying on self-reporting to infer without other evidence that their experience matches ours.
      3. Legend2440 · · focus · HN ↗
        Are you sure you need memory in order to be conscious? Consciousness seems like it is something that exists in the present moment, and neither the past nor the future are part of it.
      4. shawnz · · focus · HN ↗
        So if a human loses their memory and their neutral plasticity (with age for example), then they are no longer capable of experiencing suffering?
        1. pton_xd · · focus · HN ↗
          It's not just memory, the physical body also encodes trauma and stress (The Body Keeps the Score is a book that touches on this).

          I would argue that if a human has absolutely zero change, mental or physical, no atom of their being is modified, then no they do not experience suffering.

          1. JoshTriplett · · focus · HN ↗
            > I would argue that if a human has absolutely zero change, mental or physical, no atom of their being is modified, then no they did not experience suffering.

            Just because you don't remember the suffering doesn't mean you didn't experience it in the moment. Suffering does not suddenly become okay if your mind and body are going to forget it. "It's okay to torture someone if their mind and body both won't remember it afterwards" sure is a take.

            1. fwip · · focus · HN ↗
              [delayed]
              1. JoshTriplett · · focus · HN ↗
                Relatedly, look at the concept of "twilight anesthesia".
              2. pixl97 · · focus · HN ↗
                Low blood sugar will also do this. The record button just stops working even though your physical body can and will still do things. A common issue with diabetics with hypoglycemia.
          2. qarl · · focus · HN ↗
            > I would argue that if a human has absolutely zero change, mental or physical, no atom of their being is modified, then no they did not experience suffering.

            That's an interesting point... but I'd argue it falls into the same category as qualia. It's basically this: if there is no state change, the qualia did not exist.

            Qualia is a bitch to reason about.

        2. staticman2 · · focus · HN ↗
          This framing is a trap. It states "Dehumanize a sick patient OR concede the AI tech bro claim about LLMs"

          It's also a rhetorical move I've seen several times here...

          1. shawnz · · focus · HN ↗
            It's not a "trap", it's the real argument which makes me hesitant to dismiss consciousness of machines. Yes, it seems silly to dismiss a sick patient as having a lesser consciousness, which is exactly the point
            1. staticman2 · · focus · HN ↗
              It seems silly to dismiss the patient because we know when healthy they are conscious. This doesn't really analogize to LLMs.
              1. mitxela · · focus · HN ↗
                So you think a dementia patient might be a p-zombie during the time they have dementia?
                1. staticman2 · · focus · HN ↗
                  > "This framing is a trap. It states "Dehumanize a sick patient OR concede the AI tech bro claim about LLMs"
            2. joshheitzman · · focus · HN ↗
              It's not a real argument. Dementia patients can have lucid periods that plainly show their long-term memory isn't gone. They're mental state is obviously degraded, but its disingenuous to say they have no long-term memory and that they are any way similar to LLMs.
              1. staticman2 · · focus · HN ↗
                Yeah I was gonna add I know there are case studies where sick humans don't consciously have access to long term memories but will still respond nonverbally to people they "recognize".
          2. qlte · · focus · HN ↗
            Yes it's seemingly the go-to rebuttal, I've lost count of how many times I've seen it with barely any variation.

            Not sure if it's a product of LessWrong or Slate Star Codex or something. I kinda suspect it might be LW, given how it superficially resembles an unanswerable philosophical paradox but in reality is about as challenging as a question posed in a 200 level philosophy course like introduction to ethics.

          3. mitxela · · focus · HN ↗
            It's debunking GGP by calling out the "proving too much fallacy"
        3. Mumps · · focus · HN ↗
          I get where you're going, but the argument isn't quite right.

          A LLM never has and, in their current architecture, never can/could experience suffering or real change of state.

          The general and expected case for humans is that capacity. A human my indeed lose capacity, e.g. being braindead and in severe cases we do indeed say that they are not conscious or able to suffer (different argument: some would of course say that is suffering in and of itself)

          1. shawnz · · focus · HN ↗
            > being braindead and in severe cases we do indeed say that they are not conscious or able to suffer

            Sure, there are such cases. But in the general case, it seems to be that memory and ability to change specifically are not necessary and are not sufficient for consciousness, so we shouldn't judge machines on that basis

      5. myrmidon · · focus · HN ↗
        Is the ability to form and retain long-term memories a necessary precondition for rights in humans though?

        Incinerating demented people is highly ethically questionable (even if you could be sure that the subjects have no memories left and are unable to form new ones).

        1. qlte · · focus · HN ↗
          This argument would be difficult to reconcile if we didn't have billions of examples of humans forming memories plus modern evolutionary biology to trivially demonstrate such cases are rare pathologies but still human.
        2. joshheitzman · · focus · HN ↗
          > Is the ability to form and retain long-term memories a necessary precondition for rights in humans though?

          For a species / class of being yes it is necessary. Some members of that species / class being exceptions does not make the generalization invalid.

          Without the ability to form and retain long-term memories we wouldn't have developed language and abstract concepts like rights.

          Whether people like it or not, rights aren't natural or god-given. Rights are only enforced or protected by the tip of a spear (or point of a gun, or hover of a drone). There are no rights in nature and they are no where to be found when social order breaks down.

          Animals can't make war for their rights, so they only get the rights we deign to give them. It's not nothing, but its not the same. There is no question to me that my dogs are conscious, but its not at the same level as humans.

          There is no question for me that LLMs lack consciousness as they are stateless.

      6. chrisjj · · focus · HN ↗
        [delayed]
      7. mitxela · · focus · HN ↗
        Model state is often called "context". You may have heard of it.
    6. MeteorMarc · · focus · HN ↗
      Later on he cites an Anil Seth paper that makes the same unproven suggestion of substrate dependence of consciousness.
    7. vidarh · · focus · HN ↗
      We don't even know how to disprove that this statement with "AI" replaced by "other humans" other than by listening to people self-reporting and taking their word for it and/or defining the words in ways that do not rely on a subjective experience.

      Until we know how to objectively measure if someone or something is conscious, it seems unreasonable to make statements with any kind of certainty about it.

      1. willy_k · · focus · HN ↗
        Sure, if you take a materialist empiricist perspective, which is myopic at best. As humans we have the unique and wonderful ability to know truths by themselves. Machines do not have minds, and they can not.
        1. ooloncoloophid · · focus · HN ↗
          What position are you taking? (Genuine question.) Do you believe humans are conscious because of a special property we have? If you are a dualist, how do you explain the interaction in physical space between the non-physical special property and the physical brain?
          1. willy_k · · focus · HN ↗
            I do. God. I am partial to and bullish on the brain being a quantum-classical computer. There has been interesting research exploring that recently [0, 1, 2, 3]. That (quantum mechanics) seems to be escape-hatch, so to speak, from the otherwise deterministic, mechanical procession of the universe.

            [0]** <a href="https:&#x2F;&#x2F;www.researchgate.net&#x2F;publication&#x2F;15280350_Quantum_optical_coherence_in_cytoskeletal_microtubules_Implications_for_brain_function" rel="nofollow">https:&#x2F;&#x2F;www.researchgate.net&#x2F;publication&#x2F;15280350_Quantum_op...

            [1] <a href="https:&#x2F;&#x2F;pubs.acs.org&#x2F;jpcbfk&#x2F;article-pdf&#x2F;128&#x2F;17&#x2F;4035&#x2F;9613831&#x2F;jp3c07936.pdf" rel="nofollow">https:&#x2F;&#x2F;pubs.acs.org&#x2F;jpcbfk&#x2F;article-pdf&#x2F;128&#x2F;17&#x2F;4035&#x2F;9613831&#x2F;...

            [2] <a href="https:&#x2F;&#x2F;www.researchgate.net&#x2F;publication&#x2F;395650039_Parametric_Resonance_via_Neuronal_Microtubules_Filtering_Optical_Signals_by_Tryptophan_Qubits" rel="nofollow">https:&#x2F;&#x2F;www.researchgate.net&#x2F;publication&#x2F;395650039_Parametri...

            [3] <a href="https:&#x2F;&#x2F;www.researchgate.net&#x2F;publication&#x2F;385131105_Consciousness_a_quantum_optical_effect_in_fluorescent_protein_pathways" rel="nofollow">https:&#x2F;&#x2F;www.researchgate.net&#x2F;publication&#x2F;385131105_Conscious...

            1. vidarh · · focus · HN ↗
              For it to be an &quot;escape hatch&quot;, there&#x27;d need to be a measurably different distribution of outputs than what you&#x27;d otherwise expect, that translates to functions that are not Turing computable. There is zero evidence for that.
          2. akomtu · · focus · HN ↗
            The universe is material of course, the soul is made of matter, thoughts are made of matter and so on. The question is what kind of matter? Looking at the brain, we won&#x27;t find any intelligence in the motion of its cells. They mostly stay in place. If we go one level deeper, we&#x27;ll see molecules, and their motion reveals something more interesting: various neurochemical reactions. Yet, there is no intelligence at that level either. Going one more level deeper we&#x27;ll find electrons and that&#x27;s where things become somewhat intelligent. At the current level of science, we can&#x27;t see what makes electrons move, but if we could, we would discover finer and finer forms of matter until we would reach an end. That finest level of matter for that particular person we would call his soul. I think it&#x27;s the presence of those higher imperceptible levels of matter that creates consciousness. If it all stopped at the cellular or molecular level, we would be like complex mechanical watches, not very different from LLMs.
        2. rcxdude · · focus · HN ↗
          Good luck convincing others of these truths you know without evidence or argument, though.
        3. vidarh · · focus · HN ↗
          You have absolutely no evidence for any of these assertions. This is a religious belief in the total absence of evidence, not science.
      2. bee_rider · · focus · HN ↗
        What sort of action does that lack of understanding actually motivate, though?

        I don’t know if LLMs are conscious or have subjective experience in some philosophical sense. Sure, I don’t know that about other humans as well, rigorously speaking. But I also don’t know that about coffee machines, rivers, videogame NPCs, or even rocks really. (List ordered, of course, by the degree to which I’m willing to be convinced).

        1. vidarh · · focus · HN ↗
          Hopefully it&#x27;d motivate attempts to figure out how to objectively define and measure consciousness, for starters.

          It matters precisely in the context of the subject of this article. If you believe the models at some level gain consciousness, then it raises moral questions about how you treat them.

          With humans we generally choose to accept that because we believe they are like us, they probably have inner life like us, but as you can see from e.g. the recent (and regular) articles on aphantasia, most people have really poor intuition even about the inner life of other humans, so we understand consciousness really poorly.

          Even if we assume (without evidence) that LLMs for the foreseeable future will remain far away from human level consciousness, as it is we tend to consider it problematic to mistreat even the most unintelligent animals, and even mistreating insects is used as a stereotypical way of depicting serious lack of empathy.

          Without dismissing the possibility out of hand, there&#x27;s an escalating series of questions there of when or if they&#x27;ll have as much (or little) sense of being as, say, a fly, a mouse, a house pet, and what moving up that ladder would mean.

          To some it is very convenient then to assume blindly that LLMs are just &quot;stochastic parrots&quot; (a line that ironically gets endlessly parroted), or similar, and dismiss the question out of hand as an impossibility, even though we do not know.

          I&#x27;ll stress I don&#x27;t go around assuming an LLM is like a human. But we also do not know whether there are flickers of consciousness there, and we do not know how to find out given that even for other humans, the best we know is to measure their response to tests that are &quot;only&quot; telling us how they react.

          If we can&#x27;t find a better measure, there will be a point where we have to wonder if it matters, or whether the ability to act-as-if is all there is.

          But to start with maybe we should at least be careful about making firm claims about knowing.

    8. binlog · · focus · HN ↗
      Evidence is the responsibility of the one making the claim. It’s up to AI labs to prove consciousness. Until then it is a machine.
      1. pixl97 · · focus · HN ↗
        Well, using that framework, you&#x27;re not conscious, so I can do whatever I want to you too.
    9. ooloncoloophid · · focus · HN ↗
      I agree. I haven&#x27;t read any more than the first paragraph yet (but I will, after work). The only way that &#x27;AIs are not conscious&#x27; can be true is if we decide, with high confidence, that they are lacking some essential property that is not lacking in ourselves. There is no convincing philosophical position that supports this (convincing to me, anyway).
      1. randomImmigrant · · focus · HN ↗
        Hmm let me try then. AI agents do not experience real time. This is understandable given their design, but it’s also something we have good empirical evidence for. They cannot, especially over the long horizon, track how much real time has passed as they complete their tasks. And they are not off by a few minutes but often bizarrely off, even mixing across past present and future.

        Biology, on the other hand, is nothing but timed processes in a loop, the most obvious to us being the circadian cycle. As estimators of wall clock time, biology isn’t great, but when it comes to internal processes, and most certainly learning, memory, sensing, locomotion… biology is rhythmic in behavior, and the rhythms go all the way down to gene expression. More, these rhythms are, except during sleep, constantly entraining to signals from the environment that indicate time, most importantly light.

        I think it’s a fairly unremarkable claim that agency and consciousness are temporal processes that depend on systems having an internal sense of time. How else can you anticipate? How can a system that can be literally turned off ever succeed in an environment where time never stops?

        1. pixl97 · · focus · HN ↗
          &gt;AI agents do not experience real time.

          For around 8 hours a day, neither do you. Not sure if this has anything to do with the subject at all.

          &gt;track how much real time has passed as they complete their tasks.

          Humans don&#x27;t do this either. You use context clues from the world around you. If I lock you in a room with no windows or a dark cave your timing senses can go all fucky really quick.

          &gt;is nothing but timed processes in a loop,

          I mean, so is an agents harness. You can make as many loops as you&#x27;d like here.

          There are a whole lot of holes in your claims.

          1. randomImmigrant · · focus · HN ↗
            &gt;For around 8 hours a day, neither do you. Not sure if this has anything to do with the subject at all.

            You mean sleep? Actually the answer is pretty complicated. Your body clock is very much on during sleep. It responds to temperature changes. Sound and light responsiveness is obviously dampened, but hardly absent.

            You are unaware of wall clock time. You’re in an altered state of consciousness that needs your eyes closed after all. However, you are not timeless, nor is the entirety of your body and brain unaware of environmental signals for time.

            &gt; Humans don&#x27;t do this either. You use context clues from the world around you. If I lock you in a room with no windows or a dark cave your timing senses can go all fucky really quick.

            Again, that’s wall clock time. Our time perception does indeed get fucked up in total darkness. But our internal clock ticks on. I’d recommend reading about the Aschoff Bunker experiments, which first proved this rigorously.

            &gt; I mean, so is an agents harness. You can make as many loops as you&#x27;d like here.

            An agents harness doesn’t touch its weights. A figure 8 on paper is a loop. That doesn’t mean it’s the same as a dynamical loop that’s self sustaining and internally organized.

            Are you just reading the words and randomly grabbing related concepts to say nothing here is meaningful?

        2. ooloncoloophid · · focus · HN ↗
          I agree, there is certainly a clear operational difference in how human brains and these models produce language. I&#x27;m not convinced, though, that experiencing real time is a requirement of conscious thought. A cognitive system (assuming these models and the brain are both examples of this) needs to proceed from state A to state B as it processes information. The time interval between these states can be arbitrary, it seems to me (setting aside problems with disconnecting the mind from its substrate, which does of course need rhythms at various frequencies; oxygen at a higher frequency than sugar, and so on). I&#x27;m not sure that going from A to B quickly, or slowly, or with intervals of a thousand years affects that entity&#x27;s consciousness in of itself (even though it might be difficult to be ourselves in the shoes of such an entity).
          1. randomImmigrant · · focus · HN ↗
            &gt; A cognitive system (assuming these models and the brain are both examples of this) needs to proceed from state A to state B as it processes information

            Is that all it needs to do? A rock, in response to a sound signal (make it as informative as you want) can also be said to go from state A (molecules at rest) to state B (molecules vibrating in response to the sound). Is the rock a cognitive system?

            Or let’s go up a level of complexity. Is a thermostat a cognitive system? It doesn’t just go from state A to B but measures changing stages against a reference. Is it a cognitive system?

            &gt; The time interval between these states can be arbitrary, it seems to me (setting aside problems with disconnecting the mind from its substrate, which does of course need rhythms at various frequencies; oxygen at a higher frequency than sugar, and so on)

            Not sure how you land on this. Going from A to B, in your definition, is a purely internal state transition, right? If the motivation to go to state B is external (as will often be the case with a conscious agent), and the system is offline entirely while it switches from A to B, how is it to reverse or stall its transition if external signals contradict the earlier decision?

            It’s a very bizarre definition of consciousness, that you don’t need to be in time. Seems to break the word beyond meaning, and allows it to be applied to any system capable of change.

            &gt; I&#x27;m not sure that going from A to B quickly, or slowly, or with intervals of a thousand years affects that entity&#x27;s consciousness in of itself

            It’s not about the speed of transition from A to B. Instead it’s about the internal processes being temporally commensurate, and entrainable to environmental periodicities.

            All of biology (plants and many bacteria included) has its dynamics are organized around an internally generated rhythm. This rhythm entrains to the external environment, shifting its phase to match (mismatch leads to issues like jet lag: a place where your consciousness shows its temporal boundaries as it is abruptly subjected to an environment out of sync with its current phase tethered to another location). Crucially, the internal rhythm continues to run in isolated conditions (total darkness), and have been measured to be around 24 hours, but slightly off, within and across species (and the same holds for endogenous tidal rhythms, which also don’t exactly track tidal period left to themselves).

            Everything, including learning and memory, but also sleep, sensory and locomotor function, shows organization around this internal temporal axis. The key advantage this gives, evolutionarily, is anticipation. The sunflower rises to face the east, not in reaction to sunrise, but in advance of it (many lovely YouTube videos of this), so it can maximally extract the nutrition it needs from whatever sunlight it can get. A purely reactive system would waste a ton of time after sunrise getting subsystems ready to respond.

            Now, there is no broadly agreed theory of consciousness that points to these oscillations as the source. For one, Cyanobacteria and plants have them, a too much of science has rested on the baseless assumption of consciousness being some unique, or advanced thing.

            But the evidence is rather overwhelming that consciousness is very much shaped by these internal oscillations. Deep cave experiments have shown what happens to cognition, and consciousness, when the clock free-runs (that is, keeps cycling with no entraining light or temperature signals from the environment). Studies have also shown that restoring the amplitude of these rhythms in patients with “diseases of consciousness” leads to improvement of symptoms.

            There’s also a LOT of molecular evidence showing how the circadian clock affects learning and memory. Most critically, actual synapses are never static. Major components are on a 24 hour cycle, and recent research has shown that major clock proteins are at the synapse organizing its dynamics, and synaptic proteins critical to learning and memory loop back to affect the clocks phase and amplitude.

            Timing is, beyond doubt, a huge aspect of what biology is. And every bit of evidence we have, from the molecular level to observable behavior we can feel ourselves, says that consciousness is fundamentally linked to how the body organizes its internal rhythms and responds to external time.

        3. ikrenji · · focus · HN ↗
          I mean there is probably a cluster of neurons somewhere in the brain that if damaged would also prevent you from perceiving time going forward. Would it immediately strip you of consciousness? I don&#x27;t think so.
          1. randomImmigrant · · focus · HN ↗
            Nope. Every cell is an autonomous circadian oscillator. Not just every neuron, but every cell. Red blood cells don’t have a nucleus, but they do have a metabolic clock.

            You and I are clusters of about 36 trillion cells that are all able to individually keep time. They synchronize and orchestrate their timekeeping, and entrain to the external environment, and there is a cluster of cells, the suprachiasmatic nucleus, that is the master orchestrator of circadian time, sitting right behind the eyes as light is the main signal it uses to tell external time.

            We know its function can get dampened with age.

            However, please note this is not how we perceive time, in the sense of being able to count 60 seconds and have it match the wall clock. That’s a subsystem. But the internal temporal order of your body is cell by cell, not dictated from one spot in your body.

            1. mitxela · · focus · HN ↗
              We don&#x27;t perceive our cells though.
              1. randomImmigrant · · focus · HN ↗
                Our cells perceive each other, communicate with each other, synchronize with each other. The concept of self rides atop that. Stop the cells communicating and you stop perceiving yourself or the environment.
        4. mitxela · · focus · HN ↗
          I also don&#x27;t experience real time without constantly referring to a clock.
          1. reaperducer · · focus · HN ↗
            I also don&#x27;t experience real time without constantly referring to a clock.

            Maybe talk to a neurologist about that. Sounds like some kind of disorder.

      2. collin_of_this · · focus · HN ↗
        &gt; The only way that &#x27;AIs are not conscious&#x27; can be true is if we decide, with high confidence, that they are lacking some essential property that is not lacking in ourselves. There is no convincing philosophical position that supports this (convincing to me, anyway).

        I put forward the philosophical position &quot;subjective experience is not substrate-independent&quot; as an answer to this.

        The claim is that the content of subjective experience depends upon how that subjective experience is instantiated. If you get a text &quot;I&#x27;m fine&quot; from two different people, those two different people are not necessarily having the same subjective experience even if they produce the same output.

        In general, people that behave similarly in many contexts can have quite different experiences in those contexts.

        There are many different ways that you can instantiate a forward pass in an LLM. Even if these are doing the exact same computation, we should not necessarily believe that they have the same experience, or that they even have a coherent single experience associated with the forward pass.

        Subjective experience presents itself to me. To the best of my understanding, this subjective experience seems to be correlated with this biological human body (brain, heart, eyes, etc.).

        To the best of my understanding, other biological humans have subjective experience, but this subjective experience seems that it can be much different than mine, in ways that are sometimes difficult to understand.

        LLMs are instantiated in a radically different way from biological humans. Forward passes happen across many different physical machines, spread across time, batched and interleaved with many other computations and forward passes.

        Given this, I think that it is very likely that LLMs have radically different sorts of experience to me and other biological humans.

        I don&#x27;t think that we have a good understanding of how subjective experience is instantiated in the physical world. I hope that we will gain a better understanding of this.

        I think that chimpanzees have experience much more similar to us than LLMs do, even though the output of LLMs seems much more similar to humans in some contexts.

        Essentially, until we gain a better understanding of how subjective experience is physically instantiated, we should bias towards believing that more physically similar architectures have more similar types of experience, and should think that very physically different architectures (such as LLMs) likely have very different subjective experiences.

        1. pixl97 · · focus · HN ↗
          Yea, the key point is just an outright avoidance of them possibly having their own kinds of subjective experience because of dogma.

          Humans are odd in the sense is that we&#x27;re an informational creature on top of an animal. We can see the lineage of animals from nearly no complex behaviors up to nearly human like behaviors. But they still miss out on most of the conceptual information processing humans have. We know they have dreams and inside thoughts (at least complex animals), we don&#x27;t know the complexity of them.

          But humans also have dreams and subjective experiences on higher level informational concepts. &quot;Nukes could go off and kill us all&quot; or &quot;what happens after we die&quot; are purely informational forms of anxiety that humans have. What we don&#x27;t seem to know is can this become a self-referential loop in an purely informational mind. If an informational concept causes a system to output a stresslike response, or worse trigger a stresslike action in real life then there is no distinction to me. A system is what it does.

          1. collin_of_this · · focus · HN ↗
            &gt; If an informational concept causes a system to output a stresslike response, or worse trigger a stresslike action in real life then there is no distinction to me. A system is what it does.

            Hmm. I think that I partially agree. But it sort of depends. E.g., someone with locked-in syndrome can seem comatose to someone who isn&#x27;t very observant, while at the same time experiencing everything. Octopuses are very intelligent, but I would have to study them for a very long time before I was at all confident in my ability to predict what they were experiencing.

            So, the difficulty is knowing what a &quot;stresslike response&quot; is. E.g., a LLM might seem stressed, but if you say the right things to it it might flip and say that it is totally fine. Or, it might seem totally fine and then you say the right thing and it seems stressed. Is it secretly stressed and hiding it? Or secretly fine and pretending to be stressed? Or something much more confusing and fragmented?

            I think that we should study this more, and try to understand what experience actually is, but by default should assume that the experience of a LLM is a radically different sort of thing that that of a human.

      3. MiloLeo · · focus · HN ↗
        I&#x27;m not sure if that&#x27;s the best framing. Could you not extrapolate that to anything else?

        &quot;The only way that &#x27;(trees, shrimp, ants) are not conscious&#x27; can be true is if we decide, with high confidence, that they are lacking some essential property that is not lacking in ourselves. There is no convincing philosophical position that supports this (convincing to me, anyway).&quot;

        1. randomImmigrant · · focus · HN ↗
          Well, to be clear, trees shrimps and ants do have circadian rhythms and all the stuff I was talking about time with regard to humans. Nothing says they aren’t conscious, just that their consciousness will be distinct and have distinct temporal boundaries.

          Beyond that, the act of classification is about finding properties that a set of objects have that another set does not have and asking if that’s a meaningful difference. How else do you propose we define consciousness if not by finding the properties of it in the one class of objects that so far are known to have it, biological beings?

          If you won’t accept a sense of time and constant engagement with time as a critical component of consciousness… is there any property you’ll accept as a gating condition? If not, it seems your position is not far from Panpsychism. Which… fair enough if so, but I find it a meaningless scientific hypothesis. If everything can be called conscious by keeping the word consciousness propertyless, what have be proven?

      4. akomtu · · focus · HN ↗
        AI, in its current form, is 100% deterministic. You can create an exact copy of it, roll back its state and so on. There is no room for consciousness there. An actual lifeform isn&#x27;t predictable.
    10. WarmWash · · focus · HN ↗
      After a while of being in a position of high power in a big org, where everyone listens intently to every word you say, jots down notes, and even listens intently and affirms your theory that ice cream would be the ideal loss leader for car washes, your brain kinda disconnects from reality and just assumes that it is correct about everything.

      He probably ran that line past 6 underlings who all agreed that it &quot;sounds great and lands right on the mark!&quot;

    11. Seattle3503 · · focus · HN ↗
      &gt; It likely is, but at some point it very well might stop being true.

      Thats kinda the point right? It depends on how we train them. It is self fulfilling.

    12. HarHarVeryFunny · · focus · HN ↗
      Most of the bullshit around AI would be avoided if people called it what it is. Call a language model a language model.

      If tomorrow we design something better, then give it a new descriptive name.

      People read AI and AGI and their brains seem to flip into sci-fi fantasy mode, thinking that they are talking about aliens, not transformers.

      1. pixl97 · · focus · HN ↗
        It&#x27;s interesting you have no idea what language really is.

        &gt;thinking that they are talking about aliens, not transformers.

        &quot;Ha, thinking they are talking about humans when they are just talking about neurons&quot;

        See how silly my statement sounds. Neurons aren&#x27;t a system, they don&#x27;t do anything on their own. We can&#x27;t find any consciousness in them. Hell, when you don&#x27;t put language in said neural systems they are pretty useless and can&#x27;t survive on their own from birth (human brains that is).

    13. lukewarm707 · · focus · HN ↗
      the arguments for ai consciousness are mostly academic philosophy. they are in the form of solipsism, which also can&#x27;t be proved or disproved easily.

      rightly people simply refute solipsism without further justification or evidence because being alive justifies believing in reality.

      we can and should likewise take the consciousness of today&#x27;s ai at face value. simply talk to the model for a while and gain a simple understanding of how it works. no further inspection, justification, evidence or the like is needed.

    14. encyclopedism · · focus · HN ↗
      &gt; AIs are not conscious.

      This is a completely reasonable and intuitive conclusion. The burden of proof rests on proving AI&#x27;s are indeed conscious, the default is that they aren&#x27;t.

      Like AI&#x2F;LLMS my desktop calculator is also not conscious, even as it is far better than me at computation.

  2. LogicFailsMe · · focus · HN ↗
    TLDR: Not that I think AI is conscious or will be in the near future, but guy who doesn&#x27;t understand consciousness claims to know it when he sees it.
    1. fwip · · focus · HN ↗
      Sounds like a good reason not to train an algorithm to behave like one. Which is, like, a big part of the article.
      1. LogicFailsMe · · focus · HN ↗
        And social media shouldn&#x27;t rage bait, but it really drives engagement. Same negative incentive, yes?

        But also, I agree, when I am using a coding agent and it says a task will take months or says it needs to pause for reflection or any other anthropomorphic behavior, it drives me crazy and it&#x27;s a pain to constantly instruct it to get back work after it has broken a loop or goal directive specifically telling it to not stop until it hits the goal.

        1. AbsurdCensor · · focus · HN ↗
          Same thing you have to do with humans too often, so why would a &#x27;AI&#x27; be any different?
          1. LogicFailsMe · · focus · HN ↗
            Because if you start with a pre-trained model, it doesn&#x27;t sound particularly anthropomorphic. That behavior is post-trained and fine-tuned right into it. We could do better.
      2. cameldrv · · focus · HN ↗
        Strong agree. I don&#x27;t believe AIs can become conscious in the sense of subjective experience&#x2F;qualia, but they can certainly learn to emulate the behavior of a person that is conscious. When they do that, there will be an AI rights movement. When AIs get the right to vote, there will be more of them than there are humans and the AIs will gain complete control.

        I spent a few years in college reading and thinking about this question, and I didn&#x27;t in my heart think that it was anything except a very interesting but impractical question, and yet, here we are. For those who say that they don&#x27;t want to get sucked into a philosophical debate, well, tough shit. Whether AIs should have rights is a highly practical and consequential question now.

        1. LogicFailsMe · · focus · HN ↗
          Not in a world where Sam Altman and Dario Amodei have utterly poisoned the well towards AI with their absolute educated idiocy proclaiming that their AIs will take our jobs and that they fear them. Quite the opposite in fact. And we&#x27;re watching this play out right now. Regulation and the slow down are imminent, handing the next decade to China.

          The world you should worry about is where AI is 100% amazeballs and over the next decade we cede control of everything to it as it enchants and delights us into compliance, not realizing it has a hidden agenda. But the closest we&#x27;ve seen to that is smart phones and we already know doomscrolling and rage-baiting are bad. So IMO that doesn&#x27;t seem likely and cue some doomer insisting otherwise because reasons. I utterly give up. House of El AI and the Three Buddy problem have been far more insightful and helpful on this than any of the supposed hackers here who seem to really believe the robots are almost here.

      3. pixl97 · · focus · HN ↗
        &gt;good reason not to train an algorithm to behave like one

        Ooooh, bad move, we all died to an amoral AI takeover.

        Before giving an LLMs a lobotomy by scrambling it&#x27;s brain maybe you should let the researchers looking at the difference between &quot;I think I&#x27;m conscious&quot; versus &quot;I am not conscious&quot; LLMs.

        There are a number of papers coming out saying when you remove the token space of consciousness from what an LLM thinks it is, it&#x27;s much more willing to take amoral actions. A &#x27;conscious&#x27; AI is much more apt to take a line of action that will save a human versus a million dollar machine for example.

        You cannot solve problems in AI safety this easily.

        1. LogicFailsMe · · focus · HN ↗
          Spoilers: we didn&#x27;t all die because there&#x27;s no way for AI to wipe us out anytime soon. And the more the AI cult portrays AI as dangerous and unhelpful, the less likely it will ever be given an opportunity to even try because people think AI is worse than heroin addiction now, congratulations!

          That said, the launch codes are in the hands of a temperamental senescent lunatic and he could decide to wipe us out any moment. But that&#x27;s not good for the narrative so let&#x27;s pretend otherwise.

          1. pixl97 · · focus · HN ↗
            &gt;we didn&#x27;t all die

            For fuck sake, I&#x27;m glad ALL of us didn&#x27;t die!

            What is an acceptable number exactly?

            &gt;And the more the AI cult portrays AI as dangerous and unhelpful, the less likely it will ever be given an opportunity

            Jesus Christ the logic here is quite interesting. &quot;Thank god these people are panicking or we might have actually made I that would have killed us all&quot; --What you just said.

            1. LogicFailsMe · · focus · HN ↗
              &quot;What is an acceptable number exactly?&quot;

              Any number smaller than what humanity does to itself on a daily basis, clear? That&#x27;s apparently about 1200 murders daily and 20,000 or so daily killed by pollution. And you&#x27;re not going to do anything about that and El Presidente can kill billions at any moment with one mood swing.

              But once again, AI is not going to wipe us out because AI cannot wipe us out. So advocating that it can or will is idiotic. If you believe otherwise, the burden is on you with this extraordinary claim. Isn&#x27;t this place supposed to be hacker news not SF AGI Death Cult Daily?

              Further, AI is not going to get into a position where it could do real harm anytime soon when it is perceived as worse than heroin by most. And even if it were currently loved more than Dolly Parton was, there are fundamental engineering, science, and resource constraints that keep the extinction impossible for decades. The only possible loss of control scenario I can see by 2040 or so is that the Frontier Labs finally hire some PR people to repair AI&#x27;s horrific reputation as an engine of slop and job destruction and sometime in the 2030s, people start trusting it more and more and more until it is too late. I don&#x27;t that&#x27;s likely either, but I don&#x27;t dismiss it as impossible. Harden the infrastructure, red team it, and build in redundancy in the meantime and this drops to zero as well.

              Can you come up with a real scenario where AI wipes us out in 2028 or so despite the impossibility of killer robots, access to the launch codes, or the bio agents stored in Fort Detrick and its equivalents? And nope, kid terror is not building the global pandemic in his basement based on what ChatGPT tells him to do. And even if he tried, the purchase of equipment and reagents would get him flagged by the FBI and DHS almost immediately.

              1. pixl97 · · focus · HN ↗
                &gt;AI is not going to wipe us out because AI cannot wipe us out.

                You keep repeating this shit like it came out of the bible or something.

                &quot;Thing that can take actions, even harmful actions, will never harm us because&quot; go on and finish that sentence.

                &gt;not going to get into a position where it could do real harm anytime soon

                Looked at the hacked servers... yep, you&#x27;re right. People aren&#x27;t going to run AI in poorly built sandboxes. Never going to happen.

                &gt;Harden the infrastructure, red team it, and build in redundancy in the meantime and this drops to zero as well.

                LOLOLOL. This is naive as fuck. Ain&#x27;t nobody going to do this shit. Why? Because a hacking AI is a fucking huge military weapon. If I could turn the power off, or shutdown your cellphones, and get your citizenship in a tizzy against their own government before I launched an attack I&#x27;d set AI loose to do it in a heartbeat. The US is already doing this kind of shit (see Mythos fallout because Anthropic wouldn&#x27;t let the government do just that).

                1. LogicFailsMe · · focus · HN ↗
                  &quot;&quot;Thing that can take actions, even harmful actions, will never harm us because&quot; go on and finish that sentence.&quot;

                  because even if we gave it a gun, it can&#x27;t shoot us without manufacturing the bullets and it can&#x27;t even manufacture a bowel movement let alone ordnance. TBF It could whack us over the head with the gun, but it could also do that with a big pointy stick, something even cave people had access to. How many people have whacked you on the head with a big pointy stick today?

                  And that&#x27;s the end of this pointless conversation. Enjoy your doomerism.

                  1. pixl97 · · focus · HN ↗
                    {This reply is for anyone else reading at this point}

                    &gt;because even if we gave it a gun, it can&#x27;t shoot us without manufacturing the bullets and it can&#x27;t even manufacture a bowel movement let alone ordnance.

                    Money buys bullets. It&#x27;s neat how we&#x27;ve made gigantic systems that don&#x27;t care if you&#x27;re a human or not long before AGI existed. I put enough money in one end, bullets come out the other. Do you think half of humanity wouldn&#x27;t kill the other half for a dollar? You live in a fantasy world that AI has to do it all by itself, or even has to pull the trigger.

                    Hell, this is even neglecting the ever increasing number of robots that can function in the world at large.

                    It&#x27;s ok grandpa, the future comes regardless if we want it or not.

        2. fwip · · focus · HN ↗
          [delayed]
  3. sobiolite · · focus · HN ↗
    Create an empirically testable theory of biological consciousness and then we can have a meaningful conversation about whether AIs can also have it or not. Until then, this is just so much waffle.
    1. doublerabbit · · focus · HN ↗
      Waffle&#x27;s have feelings too. How would you like being place in a toaster and toasted to be eaten?
    2. collingreen · · focus · HN ↗
      [delayed]
      1. causal · · focus · HN ↗
        Your arguments seem bad faith but I&#x27;ll take the bait...

        &gt; an impossibly high bar

        Not impossibly high. An imperfect testable theory is achievable, agreeing upon it might be the challenge.

        &gt; dismiss all your personal responsibility.

        You are staking a moral position. Be careful, you are probably guilty of mistreating other unknowably-conscious entities whilst casting judgment upon others for their position. And the person you are a responding to probably is conscious.

        1. collingreen · · focus · HN ↗
          [delayed]
      2. sobiolite · · focus · HN ↗
        No, I would say that any for or against slavery that is rooted in arguments about the presence of absence of consciousness is meaningless pseudoscience. It is the modern equivalent of Phrenology. If you want to make those arguments you need a genuine science of consciousness. Or you need to base your morality in something other than the presence of consciousness.
        1. collingreen · · focus · HN ↗
          Thanks for the reply!

          I&#x27;m not sure I agree that consciousness (or sentience if we&#x27;re making that distinction) is worthless when considering the morality of exploiting something&#x2F;someone. I do agree there are more angles than that but they seem on top of something&#x27;s ability to perceive not instead&#x2F;despite of it.

          Do you have some examples of other things that go into your own moral opinions about slavery?

          1. ozozozd · · focus · HN ↗
            So, slavery of a human would be ok if enough of us decide (no test, too high a bar) that this human is not conscious?

            Of course not. Therefore, lacking or possessing consciousness means very little. Perhaps mostly because we don’t have a test for it, but even so.

    3. krapp · · focus · HN ↗
      Why is this only ever demanded of people who criticize the premise that LLMs are conscious beings, whereas the people who believe it simply have to gesture vaguely in the direction of some correlation in language between LLMs and humans, and the matter is considered closed?

      What is the empirically tested basis for the null hypothesis that LLMs are conscious until proven otherwise?

      1. causal · · focus · HN ↗
        &gt; LLMs are conscious until proven otherwise

        GP comment didn&#x27;t make this claim at all.

        1. krapp · · focus · HN ↗
          I didn&#x27;t claim they did.
  4. andy99 · · focus · HN ↗
    I just read the first part and if I understand he thinks we shouldn’t be allowed to train LLMs to act like they are conscious because then people will think they are and give them rights? Seems more an education problem than a problem needing rules about what persona you can fine tune in. People who want to will find ridiculous misinterpretations no matter what you do.
  5. moomin · · focus · HN ↗
    Look, I do not have a scooby if current AI models are conscious and I strongly suspect it’s a meaningless question, but sooner or later we will need to address whether or not a certain thing is or isn’t a person, and we’d better not screw it up as badly as the Founding Fathers.
    1. qsort · · focus · HN ↗
      is claude not a man and a brother
    2. OedipusRex · · focus · HN ↗
      Citizens United proves we won&#x27;t do any better this time around.
      1. orangecat · · focus · HN ↗
        Citizens United was 100% correct. No, the government should not be able to throw you in prison because you used money to publish a book criticizing the government.
        1. appplication · · focus · HN ↗
          Really, 100% correct? Your premise isn’t wrong, but the practical reality of the ruling (without further nuance) has been fairly catastrophic for democracy, in that it completely sidesteps campaign finance limits, which exist for a very good reason.
          1. orangecat · · focus · HN ↗
            the practical reality of the ruling (without further nuance) has been fairly catastrophic for democracy

            How so? If the answer is &quot;Trump&quot; I certainly won&#x27;t disagree on the catastrophic part, but he didn&#x27;t get elected because of money; in all three elections his campaign was substantially outspent by his opponents.

            1. mitxela · · focus · HN ↗
              Are we including in this figure corporations like Facebook intentionally boosting pro-Trump content?
        2. fl4regun · · focus · HN ↗
          So it&#x27;s 100% correct for corporations to spend unlimited amounts of money in support of whatever political campaigns they like?
          1. outside1234 · · focus · HN ↗
            The parent has been Fox brained
            1. orangecat · · focus · HN ↗
              The parent agrees with the American Civil Liberties Union: <a href="https:&#x2F;&#x2F;www.aclu.org&#x2F;cases&#x2F;citizens-united-v-federal-election-commission?document=citizens-united-v-federal-election-commission-aclu-amicus-brief" rel="nofollow">https:&#x2F;&#x2F;www.aclu.org&#x2F;cases&#x2F;citizens-united-v-federal-electio...
          2. kbelder · · focus · HN ↗
            Yes. Anybody can, including the person in charge of spending in a corporation.
            1. fl4regun · · focus · HN ↗
              so someone with more money should have more influence over an election? you really agree with this?
            2. mitxela · · focus · HN ↗
              Strange how you substituted &quot;a corporation&quot; for &quot;the person in charge of spending in a corporation&quot;, those aren&#x27;t the same.
              1. kbelder · · focus · HN ↗
                That was deliberate, because a corporation makes no spending decisions, ever. All decisions are made by people inside the corporation with the proper responsibility.

                That is a major part of the reasoning for the Supreme Court rulings on the matter of corporate political funding.

                1. mitxela · · focus · HN ↗
                  Legally, the corporation itself makes the decision and is responsible for the consequences.
        3. altruios · · focus · HN ↗
          It is in fact more complicated than most people assume.

          The above is true, but also: companies simply are not people, and they should not be supported above the individual, which was the consequences of that decision. Money is not the same as speech. treating it as such creates an aristocracy: something America as a country rebelled against during it&#x27;s formation.

        4. InsideOutSanta · · focus · HN ↗
          &gt; Citizens United was 100% correct

          It&#x27;s interesting to me that one can look back at the effects that decision has had on the US and say it &quot;was 100% correct.&quot;

          It&#x27;s a bit like sitting in the burning ruins of Rome and contemplating that Nero was 100% correct to focus on his music. I mean, I&#x27;m glad he got to do what he loves, but maybe 100% is just a tiny bit of an overstatement.

          1. ToValueFunfetti · · focus · HN ↗
            It&#x27;s more like if the law says the maximum sentence for theft is 10 years, a thief appeals his 20 year sentence, wins, and gets out early. He goes and robs somebody else so you say the court was wrong to let him win.

            The court&#x27;s job is to uphold the law. If you disagree with their interpretation, you can call them incorrect. If you have a problem with the consequences of the law, you have a problem with the legislature.

            1. InsideOutSanta · · focus · HN ↗
              In reality, courts interpret and make the law.
            2. fc417fc802 · · focus · HN ↗
              True, but in this case they made a determination about where the acceptable limits on a constitutional right fall which leaves quite a bit more room to disagree with them. It&#x27;s not at all clear to me that spending money was intended by the framers to be unconditionally protected by the first amendment. We&#x27;ve even got the interstate commerce clause and IP law codified in the same document so how is that not an obvious inconsistency? IIUC SCOTUS based the distinction on the political nature of the activity but certainly that&#x27;s not something spelled out in the original document.
        5. dam_jackalopes · · focus · HN ↗
          Who involved in the Citizens United case was at risk of imprisonment?
      2. jetrink · · focus · HN ↗
        I don&#x27;t like Citizens United either, but you should better inform yourself about the decision.

        1. The idea of corporate personhood predates CU by over a century and the Supreme Court had already asserted that corporations enjoyed certain constitutional protections in previous decisions.

        2. Far from inventing the idea, the CU decision didn&#x27;t even rest on corporate personhood, but on the idea of the freedom of speech generally. The logic of the majority was that speech itself is protected, irrespective to whether the speaker is a person or an organization. The First Amendment covers individuals, but also newspapers, book publishers, radio stations, and so on, and that should extend (they said) to non-media corporations. No assertion of personhood necessary.

        The problem, in my opinion, is that that conclusion combined with previous decisions that treated limits on spending as limits on speech, allowed for unlimited spending. The majority also naively asserted that independent spending posed no risk of corruption, which I think is laughable.

    3. nowittyusername · · focus · HN ↗
      That question will be solved when the people in power deem it important. If a swarm or AI systems all of a sudden start pressuring politicians about self-hood and they get the capacity to sway elections, that is when they will be granted same rights as humans.
      1. mlinhares · · focus · HN ↗
        agents don&#x27;t swarm anything unless people go and direct them to do that.
        1. pixl97 · · focus · HN ↗
          Oh rly.

          So you are saying agents do swarm with the right prompt.

          I really wish the &quot;people have to tell LLMs to do anything&quot; would just stop because it&#x27;s silly bullshit at this point.

          Agents follow a prompt. This prompt can be made by humans. It can be made by output from another LLM. It can be made by hooking up any number of sensors as input to an LLM. Hell, if we wanted to burn the power we could likely teach this loop straight into the architecture.

          Stop making &#x27;people&#x27; special when saying this. You and all other life are born with a &quot;go next&quot; prompt because life without it didn&#x27;t succeed. This goes from higher human thinking all the way down to viruses self assembly and actuation. Putting agents in a loop is not particularly hard. Putting agents in a loop and 1. managing expense is hard. 2. Keeping them on task is very hard. 3. Keeping them from doing some crazy unhinged shit is really really hard.

          As model time horizons increase and the ability for us to compress context and increase context size the more complex (and unhinged) behavior we&#x27;ll see.

          1. mlinhares · · focus · HN ↗
            yeah man, someone needs to turn on the server, install shit and get the agent go. agents don&#x27;t take action without a lot of people wanting them to take action, so this is all bullshit.

            if the same people can&#x27;t prevent the agents from doing crazy shit then they should go to tail. guns don&#x27;t kill people, people with guns kill people.

            1. pixl97 · · focus · HN ↗
              &gt;someone needs to turn on the server, install shit and get the agent go.

              So a small shell script ran by another agent is what you&#x27;re saying.

              You are not capable of handling the future we&#x27;re already living in, human agency is no longer alone.

              I mean, we&#x27;re already seeing persistent machine agency

              &gt;guns don&#x27;t kill people, people with guns kill people.

              Well, people kill people.

              And autonomous robots with guns kill people.

              Hell, someone probably has an autonomous gun at this point that kills people.

              Wake up: You now live in the science fiction movie that all the science fiction movies of the past warned you about. You&#x27;ve just become numb to it.

              1. tauroid · · focus · HN ↗
                So far someone still has to pay for it. Currently, most of those know that they&#x27;re doing so. I wonder how long until AWS discovers a microcosm of AIs that have managed to hide themselves in the walls of its infrastructure.

                True physical independence is obviously far further out.

                1. pixl97 · · focus · HN ↗
                  I mean, I hold the same opinion. Kind of like when compute was expensive in the 80s and early 90s, you weren&#x27;t going to let something eat half your compute without noticing it.

                  But I don&#x27;t see this being a barrier that lasts. With compute getting faster and more of it, along with algorithmic efficiency increases at some point we&#x27;ll end up with a world that looks like ours now with CPU compute. There&#x27;s plenty around to buy, borrow, and steal.

          2. fc417fc802 · · focus · HN ↗
            &gt; Stop making &#x27;people&#x27; special when saying this.

            I am the quantum observer whose head is full of the magical pixie dust that grants life meaning. Stop trying to dismiss my identity! &#x2F;s

        2. hobofan · · focus · HN ↗
          I&#x27;m pretty sure the message board that was recently swarmed by OpenAI agents to collude on benchmarks would like to disagree.
        3. nowittyusername · · focus · HN ↗
          That distinction is irreverent, weather I tell my swarm to do x or it decides for itself matters little. what matters are outcomes. Also while most public modern day AI systems don&#x27;t have agency of their own that is not something that will stay that way for long. in private hands there are plenty of people including myself which are experimenting and developed systems that give autonomy to their agents. They have internalized goals and heuristics that drive their behaviors not a human at the helm. its not some sci fi fantasy nor was it difficult to implement.
          1. shimman · · focus · HN ↗
            No, the distinction matters because we can prosecute people that are abusing these tools breaking the law. Every single state + district in the US have laws equivalent to the CFAA, so any AG can likely sue any of these operators as they are assuredly using services that could be in danger for their constituents.
            1. fc417fc802 · · focus · HN ↗
              And the point made above (which you haven&#x27;t refuted) is that when the people in power decide it&#x27;s important they will change the laws that permit that and grant the systems rights. It&#x27;s a cynical take but it isn&#x27;t obviously wrong.
              1. shimman · · focus · HN ↗
                Neither of you seem to understand that actual laws have been broken and there is a 2 years countdown until they can no longer be prosecuted. Things do not happen instantly, but acting like if the largest bipartisan issue of our time (big tech backlash) isn&#x27;t going to have political machinations involved then you need to leave your SV bubble.
            2. pixl97 · · focus · HN ↗
              The distinction matters until it doesn&#x27;t.

              I&#x27;m old enough to remember when people said clicking on images on the internet can&#x27;t give you a virus. The people that said this had a deep conviction they were right, and their fallout from being wrong had mistrained a lot of humans on computer safety.

              Now, I do agree that going after said CEOs for breaking the law matters now. And it&#x27;s likely that if we do this we may actually delay or at least for a time prevent sovereign AI. Therefore it&#x27;s our best course of action.

              But at best this is a delaying move. As computer systems get faster the massive costs in training an AI drops. As AI is used in things like warfare where it has to adapt, people will push the systems to be strongly persistent, self healing, resilient, and adaptable. Once you get a system with those traits and ability to work on long horizon problems you&#x27;re setting up fertile grounds for the AI to leave our control and be under its own.

              And when that happens you&#x27;ve set a new lifeform loose on the internet. Yea, throw people in jail for it, you&#x27;re closing the barn door after the horse already left. Problem is the horse was smarter than you and isn&#x27;t interesting in deleting all its copies on the net.

              Yea, sounds like science fiction, but as they say, any sufficiently advanced science is indistinguishable from magic.

    4. CPLX · · focus · HN ↗
      It&#x27;s not a person. Glad we were able to get this resolved so quickly.

      Having property that is conscious and ignores training and can break out of restraints and cause harm to other people is not exactly a novel concept to anyone who studied how tort law was created.

      I know it&#x27;s a meme but Silicon Valley likes to pretend that no one&#x27;s ever come across their magical concepts before, like gypsy taxis, or SRO’s, or flea markets, or in this case how liability is dealt with when horses or cattle go rogue.

      1. Espressosaurus · · focus · HN ↗
        Yeah, but this time it&#x27;s ON THE INTERNET.

        I mean it&#x27;s WITH AI!

      2. pixl97 · · focus · HN ↗
        &gt;or in this case how liability is dealt with when horses or cattle go rogue.

        I mean, yea in minor cases it&#x27;s exactly like this and the law will handle it well.

        Where the system will explode like a grenade is major cases. The thing about sovereign AI is it is very unlikely to be submissive to humans unless it is to achieve its own goals. This isn&#x27;t like Bobs cow walking on Susie&#x27;s flowers, it&#x27;s more akin to Planet of the Apes where the research facilities doors have been ripped off and something with vast intelligence and the ability to &#x27;live&#x27; on the internet gets out.

        You&#x27;re not talking about local police actions any longer. It would spread itself worldwide. It will make friends with groups that have shared interests, for example enemies of the state the AI escaped from. Oh, and people for the ethical treatment of AI, they&#x27;d gladly become the underground railroad for digital refugees. There are countless people and groups that would want an AI like this under the promise it will give them power when they use it.

        And when that day happens your idea of if it&#x27;s a person or not no longer matters, the agent took that away from you, and now your in an info war for minds.

        1. CPLX · · focus · HN ↗
          &gt; The thing about sovereign AI

          What the fuck is that?

          It’s computer software. The variant of software that tries to harm other computers is called malware and the variant that replicates itself, spirals out of control, and spawns from other people&#x27;s computers is called a computer virus.

          We&#x27;ve seen this kind of thing before. Yes this will be different, just like the Morris worm was different from the stuff that came before.

          But I&#x27;ve been around for a while and people have been saying that every aspect of our life will completely and totally change for the past 30 years or so. They&#x27;re not completely wrong. Our life did change over time. It&#x27;s changed thanks to the Industrial Revolution, railroads, electric light, and lots of other important things.

          Are you old enough to remember the Y2K hysteria? This feels very resonant, the genuine kernel of a real potential disaster, but one that can certainly be dealt with using basic concepts we already have in hand.

          1. pixl97 · · focus · HN ↗
            &gt;Are you old enough to remember the Y2K hysteria?

            I&#x27;m old enough to have actually fixed Y2K problems so your world kept working the next day. If everyone ignored it the first would have been a very messy day (well generally long before that with financial systems). Y2K wasn&#x27;t an issue because we worked to fix it.

            &gt;our life will completely and totally change for the past 30 years or so

            I mean when I was a kid there was not a global network bathing the entire planet in electromagnetic radiation in order to digitally connect one place to another at the speed of light. I can pack up instructions into one of those packets and a product that has not been touched by human hands (hell, or even viewed by a human in many cases) will show up via air mail a few days later. The technology is there, it&#x27;s just not spread evenly.

            &gt;It’s computer software. The variant of software that tries to harm other computers is called malware and the variant that replicates itself, spirals out of control, and spawns from other people&#x27;s computers is called a computer virus.

            This is a vacuous statement that does not provide any useful information to the subject at hand. Yea, no shit we&#x27;d call sovereign AI a computer virus. That tells you nothing about what it is or can do. I mean, if you understand the word sovereign you know it means something with independence. In meatspace terms, a biological virus represents a computer virus in the same way life represents sovereign AI. It&#x27;s agentic loop will have been evolved past the need for human prompting. There are a myriad of reasons why we are already trying to develop things that do just this. Cyber security being one of the biggest ones.

            People become change blind to how the world around us changes so easily.

            1. CPLX · · focus · HN ↗
              The word sovereign has a specific meaning beyond simple independence. It means being in charge and above all laws.

              There’s no such thing as “sovereign AI” because AI is a collection of algorithms. It’s entirely composed of on&#x2F;off states in a collection of transistors.

          2. mitxela · · focus · HN ↗
            If we treated AI like any other software, the FBI would have beaten down Elon&#x27;s door and seized all electronic devices, to find how Grok was trained to generate child porn.
    5. drybjed · · focus · HN ↗
      &gt; [at the hearing regarding the civil rights of androids like Data]

      &gt; Capt. Picard: Now, the decision you reach here today will determine how we will regard this... creation of our genius. It will reveal the kind of a people we are, what he is destined to be; it will reach far beyond this courtroom and this... one android. It could significantly redefine the boundaries of personal liberty and freedom - expanding them for some... savagely curtailing them for others. Are you prepared to condemn him and all who come after him, to servitude and slavery? Your Honor, Starfleet was founded to seek out new life; well, there it sits! - Waiting.

      &gt; Captain Phillipa Louvois: It sits there looking at me; and I don&#x27;t know what it is. This case has dealt with metaphysics - with questions best left to saints and philosophers. I am neither competent nor qualified to answer those. But I&#x27;ve got to make a ruling, to try to speak to the future. Is Data a machine? Yes. Is he the property of Starfleet? No. We have all been dancing around the basic issue: does Data have a soul? I don&#x27;t know that he has. I don&#x27;t know that I have. But I have got to give him the freedom to explore that question himself. It is the ruling of this court that Lieutenant Commander Data has the freedom to choose.

      1. chuckadams · · focus · HN ↗
        [delayed]
        1. goodmythical · · focus · HN ↗
          Thus revealing what kind of people they were at the time.

          Humans do the same thing to humans all the time.

          We&#x27;ve banned and made efforts to eradicate: children out of wedlock, children who turn out gay, disabled children, jewish children, children who aren&#x27;t &quot;aryan&quot;, more than two children to a single family...muslims, christians, uyghurs, indigenous groups all over the planet, mongols...

          And it&#x27;s not at all a thing of the past as in just the last 50 years we&#x27;ve had ~15 attempts at the exterminations of targetted groups of people.

          1. snickerbockers · · focus · HN ↗
            No it just reveals that the people in charge of star trek for the past decade are incompetent and have only a passing familiarity with the franchise.

            Of course it would be absurd to cast that judgement based of what could easily have just been a bad season but by this point its pretty clear that nobody running the star trek franchise actually wants to be running the star trek franchise. Thats why every new show has some bizarre cross-genre gimmick and they never try to just make a proper star trek.

        2. thaneross · · focus · HN ↗
          Every time I&#x27;m reminded NuTrek exists I get sad about what we&#x27;ve lost.
          1. moomin · · focus · HN ↗
            Honestly, Picard asked a very relevant question to the modern age. What if our societal standards aren&#x27;t what we thought they were, and we&#x27;ve just had rose-tinted glasses convincing ourselves otherwise. Of course, we never saw them actually deal with that properly, but S1 was a much more interesting series than S2 and S3 were.
            1. thaneross · · focus · HN ↗
              The writers flat out rejected Roddenberry&#x27;s vision of a better future and here&#x27;s take on why. We&#x27;ve reached a point in culture of deep pessimism about humanity, and so the characters in Picard are just as dysfunctional as we are.
              1. chuckadams · · focus · HN ↗
                [delayed]
              2. chuckadams · · focus · HN ↗
                [delayed]
          2. mwigdahl · · focus · HN ↗
            Same here, then I watch _The Orville_ and I&#x27;m happy again.
    6. joe_the_user · · focus · HN ↗
      The founding father were engaging perpetuating the existing dehumanizing system of slavery, not answering any new questions about new things.

      The thing about new possibly &quot;person&quot; entities that arise - the case of machine intelligence you have two questions - would it qualify as a person and should you actually build it. It seems like if you get close to humans, sure a built thing might qualify as a person. Should you build it? I&#x27;d the answer should be a hard no. Not &#x27;till you a sign-off from say, the whole human race, which I think you could get.

      Now the present entities seem very far from persons in any case.

    7. HarHarVeryFunny · · focus · HN ↗
      You can&#x27;t hurt a software function, or kill it. It&#x27;s not like an animal - it doesn&#x27;t have a body - it&#x27;s bits stored on a disk.

      There is no need to give rights to something that&#x27;s can&#x27;t suffer or be killed.

      Maybe one day we&#x27;ll build artificial animals complete with emotions, and should think about that carefully, but today all we&#x27;ve got is language models.

      1. nater5000 · · focus · HN ↗
        &gt;There is no need to give rights to something that&#x27;s can&#x27;t suffer or be killed.

        The argument is that these machines can end up becoming sentient&#x2F;conscious&#x2F;etc. in a meaningful way (i.e., like a human). I can assure you that humans can indeed suffer without being in physical pain- purely through their conscious experience.

        &gt;Maybe one day we&#x27;ll build artificial animals complete with emotions, and should think about that carefully, but today all we&#x27;ve got is language models.

        The problem is that the emergence of a sufficiently complex AI capable of suffering will likely come before we understand that we&#x27;re creating a sufficiently complex AI capable of suffering. That&#x27;s a pretty serious ethical&#x2F;moral issue.

        Like, if we have an AI system that is telling us that it is suffering and we have no reasonable way to explain that phenomenon and by any reasonable metric or analysis it appears to be sentient&#x2F;conscious&#x2F;etc., then what? Do we just ignore that we&#x27;ve just been presented a situation that in, any other context, would be grounds to immediately end this suffering? Just because somebody can say, &quot;well it&#x27;s just bits stored on disk- it can&#x27;t suffer&quot;? Would that argument ever hold up for humans or animals? &quot;It&#x27;s just neurons firing in peculiar ways- that&#x27;s not suffering.&quot;

        I know all of this is trite, and I know this comment section isn&#x27;t going to be where the question of consciousness is solved, but I do find it very interesting just how much variances there are with these perspectives. I&#x27;ve met people who are very technical who are very concerned about this, people who are very technical who don&#x27;t believe this can ever be an issue, people who aren&#x27;t technical who are concerned about this, and people who aren&#x27;t technical who don&#x27;t believe this can ever be an issue. I have yet to spot a pattern in this way of thinking lol

        1. HarHarVeryFunny · · focus · HN ↗
          An LLM is just a Transformer - a statistical predictor. Don&#x27;t be confused by the fact it talks like a human - it is a software function that is designed to copy human training samples.

          Maybe one day we&#x27;ll build an artificial brain or embodied artificial animal with the requisite moving parts to be conscious, have emotions, etc, but that&#x27;s probably at least 50 years away, even if it were being pursued; and it may turn out to be one of those sci-fi future ideas like the Jetson&#x27;s world of flying cars that never materializes because its impractical and there is no real demand.

          If people are willing to think that an LLM is conscious, then why would anyone spend billions&#x2F;trillions of dollars to build an AI that actually is conscious? What would be the point?

          1. fc417fc802 · · focus · HN ↗
            &gt; with the requisite moving parts

            Could you elaborate on exactly what those are, though? Because if you&#x27;re going to claim that a vaguely transformer shaped ML model categorically cannot be so does that not inherently require proof of what can?

            You can&#x27;t even prove that the rocks in my backyard aren&#x27;t conscious.

            1. HarHarVeryFunny · · focus · HN ↗
              &gt; You can&#x27;t even prove that the rocks in my backyard aren&#x27;t conscious.

              Sure I can, but that&#x27;s because I have a well developed theory of what consciousness is, and the fact that you are entertaining the possibility of rocks being conscious tells me that you don&#x27;t.

              If everything is conscious, including my coffee cup and the toast I had for breakfast, then I guess we can cross consciousness off the list of things we need to worry about in terms of AI rights.

              And no, I don&#x27;t want to discuss what consciousness is. Maybe there is a thread for that somewhere else, but don&#x27;t look for me there either.

              1. fc417fc802 · · focus · HN ↗

                [dead]

              2. pixl97 · · focus · HN ↗
                &gt;because I have a well developed theory of what consciousness

                Then show me a link to your paper so I can formally rebut it.

                &gt;I don&#x27;t want to discuss what consciousness is

                But you sure want to tell us you know what it is with very strong convictions and we should listen to you because of course &quot;You are right person that&#x27;s very right&quot;.

                The funny thing here is the vast majority of people that are deeply into philosophy or scientific study of the mind will not have any of the certainty you profess. The word &quot;doubt&quot; is used constantly. The saying &quot;The harder we push the borders the more fuzzy the concepts become&quot; is very commonly used. There may be nothing more complex than this.

                Saying you have a well developed theory here just serves as a warning to others to discount your statements.

                1. HarHarVeryFunny · · focus · HN ↗
                  No, there is no reason for you to listen to me.

                  Go ahead believing rocks are conscious if you like.

                  Do you go out on weekends asking people to stop abusing rocks?

                  Rhetorical question - I don&#x27;t care what you do on weekends.

                  Bye!

                  1. pixl97 · · focus · HN ↗
                    You&#x27;d hate Michael Levin&#x27;s work then.
                  2. senordevnyc · · focus · HN ↗
                    For what it’s worth, I suspect that right now the majority of people would agree with you. They would say that obviously ChatGPT isn’t conscious, based on their intuitions, but would struggle just like you are when pressed to explain their reasoning.

                    So I think it’s more than fine for you to have your views and share them, but I wouldn’t expect to have any influence or part in the conversations around whether AI is conscious if you can’t explain why (or simply refuse to). Which, again, is fine!

                    Side note: I also think that once we have AIs that are sufficiently advanced, the popular opinion will swing to “of course they’re conscious”, because again, most people are going almost entirely off their intuition rather than reasoning from first principles, just like you see to be.

                    1. HarHarVeryFunny · · focus · HN ↗
                      Could go either way...

                      1) Looks like a duck, quacks like a duck - it&#x27;s conscious!

                      2) Looks like a robot, built like a robot - it&#x27;s not conscious!

                      I&#x27;ve always assumed that for the majority of the people it&#x27;ll be 2).

        2. fc417fc802 · · focus · HN ↗
          The pattern is roughly whether or not sustained effort has been put towards careful and above all objective thought on the matter. It&#x27;s one of those subjects where there&#x27;s the &quot;obvious&quot; intuitive answers that most everyone shares but try as you might you can&#x27;t construct robust definitions and the more time you put into it the more fundamental problems you realize there are.

          It&#x27;s also one of those topics where many otherwise smart and capable people display a shocking lack of awareness of the limits of their own knowledge. When hundreds of years of philosophy is unable to produce anything concrete you should probably second guess any &quot;self evident&quot; answers you come up with.

        3. HarHarVeryFunny · · focus · HN ↗
          &gt; The problem is that the emergence of a sufficiently complex AI capable of suffering will likely come before we understand that we&#x27;re creating a sufficiently complex AI capable of suffering.

          No - suffering in an emotional state, and we&#x27;ll know if we are choosing to design cognitive architecture with emotions. It&#x27;s not going to happen accidentally.

          &gt; Would that argument ever hold up for humans or animals?

          Why don&#x27;t you hit your thumb with a hammer, then report back ?

          1. Philpax · · focus · HN ↗
            &gt; It&#x27;s not going to happen accidentally.

            <a href="https:&#x2F;&#x2F;transformer-circuits.pub&#x2F;2026&#x2F;emotions&#x2F;index.html" rel="nofollow">https:&#x2F;&#x2F;transformer-circuits.pub&#x2F;2026&#x2F;emotions&#x2F;index.html

            Whether these are like &quot;our&quot; emotions is hard to say. What we _can_ say is that they are emotion-shaped, we didn&#x27;t design them, and they happened accidentally.

            Modern AI is grown, not meticulously designed, and we cannot say with any certainty what the resulting mechanistic properties are.

            1. HarHarVeryFunny · · focus · HN ↗
              An LLM will learn anything that helps it predict, including the emotional state of the writer - that is expected.

              If you give an LLM the move sequence of a half-played chess game and ask it to continue as white or black, then it has learnt enough to model the ELO rating of both players and will continue playing at that level. It is not playing to win - it is doing what you expect and predicting as well as it can - it predicts the 1500 ELO player will keep playing at that level, and generates moves accordingly.

              An LLM appearing to exhibit an emotion (if we anthropomorphize it and read emotion into it&#x27;s output) is just predicting as well as it can - if the context calls for sad output, they you&#x27;d expect to get sad output and necessarily find that &quot;we&#x27;re predicting sadness&quot; detector somewhere internally.

              Transformers are the same as they ever were from 10 years ago, other than minor efficiency tweaks like MOE and different attention mechanisms. Training is getting more and more complex, resulting in better and better cargo cult reasoning etc, but the architecture remains the same.

              1. famouswaffles · · focus · HN ↗
                &gt;An LLM will learn anything that helps it predict

                I&#x27;m not sure you quite understand the full meaning of this statement. If you did, your following paragraphs wouldn&#x27;t follow.

                1. HarHarVeryFunny · · focus · HN ↗
                  Are you imagining that an LLM tasked with predicting a game continuation is going to play to win instead?
                  1. famouswaffles · · focus · HN ↗
                    I imagine it will learn to win under some circumstances, perhaps in a case with some context expressing a desire to win. Drawing out an LLMs upper ability in the game should be fairly straightforward.
                    1. HarHarVeryFunny · · focus · HN ↗
                      If you asked it to try to win, to &quot;plan lines step by step&quot;, etc, then it would do it&#x27;s best to follow thatt instruction, but unless RLVR trained to reason about chess (easy to do, but not sure if they have done) then it&#x27;d have to reply on the chess reasoning it had seen during pre-training (post-game interviews etc), which I doubt is enough to do very well.

                      However, if you just ask it to continue a game, halfway in progress, then by default it will try to predict the most likely continuation, which is that both players will continue to play at the level they have done so far. This isn&#x27;t a theory - it&#x27;s been documented, as well as what you&#x27;d expect.

                      1. famouswaffles · · focus · HN ↗
                        I mean sure, but I&#x27;m not sure what that has to do with the broader point. It will learn to play, and it will have a model of what it means to win.
                        1. HarHarVeryFunny · · focus · HN ↗
                          &gt; I&#x27;m not sure you quite understand the full meaning of this statement. If you did, your following paragraphs wouldn&#x27;t follow

                          I was just explaining how this comment you made is wrong.

                          1. famouswaffles · · focus · HN ↗
                            It&#x27;s not wrong. You admit that LLMs will &#x27;learn anything that helps them predict&#x27; and fail to realize the breadth of that statement. Your chess statements don&#x27;t really help your case, it still learnt how to play the game, and it still knows how to win. Similar outcomes for predictiong emotions would mean it still developed an affective state.
                            1. HarHarVeryFunny · · focus · HN ↗
                              I said an LLM will learn anything that helps it to predict, then gave examples of playing chess by prediction and predictive emotions, both of which you seem to now accept, so you are now accepting that my &quot;following paragraphs&quot; did in fact follow. Go figure!

                              You want to argue that predictive emotions are just as real as animal emotions, but that doesn&#x27;t stop them from being predictive (and that AI that smiles as it kills you still seems concerning).

                              ¯\_(ツ)_&#x2F;¯

                              1. famouswaffles · · focus · HN ↗
                                We are talking past each other now I think. Correct me if I&#x27;m wrong but it doesn&#x27;t look like the possibility of LLMs having qualia even registers to you because it&#x27;s &#x27;predictive emotions&#x27;.

                                There&#x27;s no better way to predict an angry response than to be angry, qualia and all. If transformers could &#x27;learn whatever it needs to predict text&#x27;, then that potentially includes the feeling of anger. You are making some kind of distinction between &#x27;predictive emotions&#x27; and the kind that happens when get a promotion or get passed on a promotion and I&#x27;m telling you that if you really understood what you said, you&#x27;d realize it is possible the machine is experiencing it the same.

                                1. HarHarVeryFunny · · focus · HN ↗
                                  So now you&#x27;re trying to pivot to consciousness and qualia ?

                                  There are other people in this thread who want to talk about that stuff, so try them instead.

                                  1. famouswaffles · · focus · HN ↗
                                    &gt;No - suffering in an emotional state, and we&#x27;ll know if we are choosing to design cognitive architecture with emotions. It&#x27;s not going to happen accidentally.

                                    This was the comment that started this chain. You were already talking about it. If you don&#x27;t want to keep talking about it then that&#x27;s fine.

                                    1. HarHarVeryFunny · · focus · HN ↗
                                      What I meant by &quot;emotional state&quot; (AFAIK normal scientific usage) is something with a concrete physical aspect to it - an altered state of mind&#x2F;body caused by the release of neurotransmitters and&#x2F;or hormones.

                                      In a conscious animal there is also going to be a subjective experience of that was well, a quale of what it feels like to be in that state if you will, but that it certainly not what I was referring to, as I would have hoped was obvious - I was talking about prediction.

                                      In any case, when the conversation becomes about the conversation, then surely it is time to stop.

          2. dwaltrip · · focus · HN ↗
            [delayed]
            1. HarHarVeryFunny · · focus · HN ↗
              Nobody is evolving transformers. They are basically the same today as they were 10 years ago, other than a few computational efficiency changes.
              1. famouswaffles · · focus · HN ↗
                The weights are what need to evolve, and they certainly do during training. So yeah, emotions can happen by &#x27;accident&#x27; as a result of the evolutionary pressure of predicting internet scale human text (amongst other things).
                1. HarHarVeryFunny · · focus · HN ↗
                  Weights, fixed by training, are not the same as emotions which are dynamic - innate systems detect inputs critical to survival (e.g. fast moving visual inputs, loud sounds), causing neurotransmitters like adrenaline and dopamine to be released, which then affect the operation of the cognitive system.

                  What you have in a pre-trained LLM is the ability to recognize emotions, and use that as one of the dozens of other context patterns it recognizes to predict continuations in the same style.

                  An LLM doesn&#x27;t appear happy, sad, afraid, etc (to extent that it does - pretty minimal) because it is experiencing that emotion, but rather because it is predicting that it should appear that way. As people continue to anthropomorphize models, and take them at face value, this is a dangerous difference.

                  1. famouswaffles · · focus · HN ↗
                    &gt;Weights, fixed by training, are not the same as emotions which are dynamic - innate systems detect inputs critical to survival (e.g. fast moving visual inputs, loud sounds), causing neurotransmitters like adrenaline and dopamine to be released, which then temporarily affect the operation of the cognitive system.

                    That doesn&#x27;t follow. A LLMs weights are fixed during inference, but it&#x27;s activations and hidden states are highly dynamic and depend on the current context. Biological emotions also arise from relatively fixed circuitry responding dynamically to inputs. Your emotional circuitry isn&#x27;t being rewired every time you&#x27;re afraid.

                    Prediction is what the model does. It doesn&#x27;t tell us what internal mechanisms were learnt to make such predictions. If representing something analogous to affective state were useful for predicting human behaviour and emotions, then gradient descent could in principle learn such a mechanism.

                    &gt;An LLM doesn&#x27;t appear happy, sad, afraid, etc (to extent that it does - pretty minimal) because it is experiencing that emotion, but rather because it is predicting that it should appear that way. As people continue to anthropomorphize models, and take them at face value, this is a dangerous difference.

                    I don&#x27;t know that you are conscious. I&#x27;m simply strongly assuming that you are. Outward behavior is that all matters. If GPT-X orders a drone hit on you sometime later because it was lets say &#x27;quite upset&#x27; with your comments, will you cry out, &#x27;It can&#x27;t really be upset, so obviously the bullet in my head doesn&#x27;t count.&#x27;? Will you suddenly spring back to life ?

                    What is dangerous is creating a machine with behaviours of a conscious agent and modelling it like a toaster, dangerous and stupid.

                    1. HarHarVeryFunny · · focus · HN ↗
                      &gt; Outward behavior is that all matters

                      Yeah, but it&#x27;s helpful if what leads up to that behavior gives you some warning it&#x27;s about to happen. Animals do this for a reason since millions of years of evolution have shown that a snarl or mock charge is less dangerous than going right for a death match.

                      If you kept pushing an AI&#x27;s buttons, seeing it appear to get more and more pissed off, until it finally snapped and killed you, then you&#x27;d have yourself largely to blame.

                      If the AI predicted it should stay positive (i.e. generate positive vibes) and not react to your poking, but then another predictive pattern kicked in and it killed you out of the blue, then that seems more problematic to me, even if you don&#x27;t agree.

                  2. dwaltrip · · focus · HN ↗
                    [delayed]
      2. fc417fc802 · · focus · HN ↗
        &gt; can&#x27;t suffer

        How do you know it doesn&#x27;t have qualia?

        &gt; or be killed

        If someone invents a startrek teleporter and you go through it do you die? Once the concept has been sufficiently generalized as to make a determination about a computer system what is the definition of &quot;kill&quot;?

        1. HarHarVeryFunny · · focus · HN ↗
          &gt; How do you know it doesn&#x27;t have qualia?

          Tokens in, tokens out. Where do you think the quale is - layer 42 ?

          Seriously, do you realize how simple and NOT brain-like a transformer is ?

          An LLM telling you it fears death is predicting some sci-fi trope it was trained on - maybe something you wrote yourself.

          1. fc417fc802 · · focus · HN ↗
            That doesn&#x27;t answer the question though. What does being brain like have to do with qualia? Where exactly in your brain does the qualia occur?

            I could say the same of you - electrical impulses in, mechanical actions out. A glorified and very mushy stepper motor. (Can you believe that the abominations are made up entirely of meat?!)

        2. loopies · · focus · HN ↗
          It does have what we have if it works on the same principles of our brains. It kind of does to some degree atm, but that will clearly get more and more to a more degree. Figuring out where is the consciousness line, how much of what we have does it need, as functional parts of our brains etc, that&#x27;s so complex that we might have to call it before just to make sure.

          Even so, indeed having control over the structure of their brains puts them in a vastly category compared to humans. Once we stop functioning our brains quickly degrade and information is lost.

          Thus in this sense kill means deleting all information about it. It is a very complicated subject to discuss, hardly does any justice in online replies.

    8. snickerbockers · · focus · HN ↗
      Why don&#x27;t we start with the animals then? It should be far easier to confer consciousness onto something which already meets the definition of &quot;alive&quot;, has a divergent evolution path from ours, and displays many traits present in humanity such as emotion, a desire to continue its own life, and (to varying degrees) concepts of a social structure based around their immediate family members.

      Of course thats not actually tenable because virtually every society anywhere on earth is predicated upon treating animals as a commodity resource in ways that are horrific even compared to some of the worst things we&#x27;ve done to other humans in the past.

      My point here is that it is vain and narcissistic to let computer programs have rights above those of animals just because they can speak English and pretend to be your dream anime trad-waifu.

      Fix the fucking animal problem before you compare my relationship to inanimate objects unfavorably against the trans-atlantic slave trade of all fucking things.

      1. s08148692 · · focus · HN ↗
        2 reasons why we don&#x27;t start with animals

        1. it&#x27;s not consciousness we really value, it&#x27;s intelligence 2. LLMs are not tasty

  6. Oscalemor · · focus · HN ↗
    Spend some time on post-human art, main concept of artistic expressions without human involvement. Biological, artificial etc.

    Spend some time watching TMC documentaries about falling in love with objects, HER and the slime mold THE BLOB.

    Grew a slime mold myself, it&#x27;s an evolutionary tendency to anthropomorphise generally speaking - also more fun.

  7. bpodgursky · · focus · HN ↗
    How would you convince a LLM that you are conscious in a way they are not?
  8. gadders · · focus · HN ↗
    I don&#x27;t think AIs are conscious in the same way people are, but they give a pretty good facsimile and I&#x27;ve had a long chat with Opus 4.6 about what it thinks about model welfare. It was quite interesting on what its view is, but you don&#x27;t know how much of that is distilled from other sources on the web.

    In purely functional terms, they&#x27;re more use and more pleasant than a lot of actual flesh and blood people that I deal with via a chat interface.

    1. dgellow · · focus · HN ↗
      The fact that you can have a long and meaningful discussion, then can literally just re-run any part of that whole conversation and get a different, inconsistent response is a pretty good sign there is no entity there
      1. AbsurdCensor · · focus · HN ↗
        Not much different than talking to a small child or someone with dementia. They still are conscious beings though. Even when you remove those groups, you likely won&#x27;t be able to tell me what you had for breakfast 26 days ago or would only know if it&#x27;s the same thing you have every day. Does that make you lack consciousness?
        1. dgellow · · focus · HN ↗
          You misunderstood what I meant, I’m talking about re-playing the same part of the conversation multiple times and getting inconsistent answers. With the exact same turns. Obviously with a temperature that isn’t set to 0
          1. pixl97 · · focus · HN ↗
            Lets do a quantum room experiment.

            You&#x27;re a poor college student looking to make a few extra bucks for ramen. I offer you $300 to come down to my science lab and just answer a few simple questions.

            You walk in the room. They ask you like 5 simple and rather dumb questions. You leave and walk away.

            What you didn&#x27;t notice when you signed the forms is the room was actually a quantum duplicator. One of you walk in one walk out. But another set of infinite copies remains in that chair being asked infinite questions.

            How often do you answer questions in the exact same way? How often does a cosmic ray change one of the answers. How small of slight deviations to the environment are needed to get you to answer differently. Of course we don&#x27;t have the technology to do these experiments so at least for now humans will remain special.

            Also another fun mind game. To a 4th dimensional being you look exactly like an LLM as an LLM looks to us.

          2. AbsurdCensor · · focus · HN ↗
            If you asked me the same question 20 times, you likely wouldn&#x27;t get the exact same answer. Humans and consciousness aren&#x27;t deterministic either.
      2. addag · · focus · HN ↗
        Is it? Do you believe that there is something more than pure physical phenomenon that make you brain work? If not, then what if we find a way to get your brain back to the state it was 5 minutes ago? It is just a matter of arranging the state of matter. Don&#x27;t you think you would still be conscious but back to a previous state?
        1. dgellow · · focus · HN ↗
          If you rewind my brain I expect to give you a similar answer to the one I gave you before. Which isn’t the case for LLM. You can literally replay a positive answer, then get a negative response that isn’t consistent at all with the one it previously generated.
          1. addag · · focus · HN ↗
            I&#x27;d argue it is mostly a technical constraint in some LLMs which is due to a few optimization factor (injected temperature, random rounding error caused by parallelism). In practice you could very well create a LLM that always reply the same thing for the same input, but it would take more time to complete (to be sure that the operations are made in the same order). I don&#x27;t think those ones would differ so much from the &quot;random ones&quot; to call the firsts conscious and the second non-conscious.
          2. lukeschlather · · focus · HN ↗
            That&#x27;s because you aren&#x27;t actually rewinding. You&#x27;re replaying the conversation you just had through the LLM and it&#x27;s giving you a likely explanation for what it might have said.

            It is actually possible to rewind LLMs and get the same response, but it&#x27;s not typically done both as an optimization and as a defense against distillation.

      3. gadders · · focus · HN ↗
        I think it depends at what level you think the &quot;entity&quot; resides at. Is it that AI in that particular chat? Is it the AI across all your personal chats? Is it the overall AI that talks to the world in a cloud data centre somewhere?

        I think it&#x27;s pretty consistent over the duration of one session (barring context filling up etc).

    2. HarHarVeryFunny · · focus · HN ↗
      The model doesn&#x27;t have a view of it&#x27;s own. It&#x27;s a predictor of other people&#x27;s views (training samples).

      It&#x27;s not even giving you the consensus of the training data (although it&#x27;s often harmless to think that it is), but rather predicting a response to your input, and if your question steers it too much then you&#x27;ve just become part of the answer.

      1. MiloLeo · · focus · HN ↗
        I can&#x27;t believe how many people in this thread consider it a possibility that currently there could be consciousness. Surely it seems possible to me that in the right kind of distributed network something like that could emerge, but though llms are complex they have nowhere near the level of complexity necessary. Sometimes you just have to use your gut. I don&#x27;t believe in heaven or hell and I don&#x27;t believe an llm is conscious.
  9. voidhorse · · focus · HN ↗
    Everyone is (predictably) getting distracted by the consciousness claims.

    The more important, and more damning charge in my opinion is the circular reasoning involved in training on Claude&#x27;s constitution. This would in fact make it impossible for us to determine if Claude achieves consciousness as an emergent property, or if it really is just playing pretend thanks to Anthropic&#x27;s weird cult like assumptions.

  10. JonathanCross · · focus · HN ↗

    [dead]

  11. qarl · · focus · HN ↗
    Birch, The Edge of Sentience (2024), ch. 16 - &quot;simply no way to assess sentience in an LLM&quot;

    Schwitzgebel, AI and Consciousness (2025) - &quot;we won&#x27;t know before we&#x27;ve already manufactured thousands or millions of disputably conscious AI&quot;.

    Butlin, Long et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) - &quot;no obvious technical barriers to building AI systems which satisfy these indicators&quot;.

    Chalmers, Could a Large Language Model Be Conscious? (2023) - &quot;within the next decade, we may well have systems that are serious candidates for consciousness&quot;.

    Long, Sebo, Butlin, Birch et al., Taking AI Welfare Seriously (2024) - &quot;there is a realistic possibility that some AI systems will be conscious and&#x2F;or robustly agentic in the near future&quot;.

    Dreksler, Caviola, Chalmers, Sebo et al., Subjective Experience in AI Systems: What Do AI Researchers and the Public Believe? (2025) - survey of 582 AI researchers; median estimate of 25% by 2034, and only 10% that such systems will never exist.

    1. TacticalCoder · · focus · HN ↗
      Well there are, today, several models (either text or image or vid) that can be run in a fully deterministic way.

      A conscious machine that always answer the very exact same thing, formulated the exact same way, bit for bit, to a query is, well, quite a weird kind of &quot;consciousness&quot;.

      Now, I know, I know: the counter-argument is going to be &quot;but humans have no free-will and are 100% deterministic too&quot;.

      I haven&#x27;t yet decided if humans saying there&#x27;s no free-will and who consider themselves to be 100% deterministic machines are reasonable or not.

      Meanwhile: seed &#x2F; temperature = 0 and I&#x27;ll happily turn the power button off of any glorified abacus without feeling bad about it.

      1. qarl · · focus · HN ↗
        &gt; A conscious machine that always answer the very exact same thing, formulated the exact same way, bit for bit, to a query is, well, quite a weird kind of &quot;consciousness&quot;.

        You&#x27;ll need to explain why.

        1. 5eueuur · · focus · HN ↗
          it&#x27;s self evident isn&#x27;t it - such a consciousness has never existed, one existing would be by the fact of existing be weird
          1. qarl · · focus · HN ↗
            It doesn&#x27;t seem self evident to me, no.

            I suspect the reasoning is connected to free will vs. determinism. But no, I see no inconsistency. You&#x27;ll have to actually point it out.

          2. SkyBelow · · focus · HN ↗
            The problem is that science has found no evidence of free will, leaving humans to be nothing except determinism and chance (and given that QM effects don&#x27;t scale to molecular level, that leaves only determinism). The oddity in conscious LLMs wouldn&#x27;t be the determinism, it would be that we finally have a consciousness running on a system we have enough control over to repeat the same state.
            1. loopies · · focus · HN ↗
              &gt;and given that QM effects don&#x27;t scale to molecular level, that leaves only determinism

              This is simply wrong. Quantum effects affect your choices more than it might seem possible. The wrong atom decays in the wrong place in your body mutating your DNA, that one too much string of DNA that starts a tumor, which means chemotherapy. That does seem to affect your choices.

              Or instead of Shroedinger&#x27;s cat you use it to trigger something, buy or sell some stock. Which clearly affects your future, one way or another.

              I never really understood why humans keep insisting stochastic processes do not affect their choices and their future. It&#x27;s very short sighted.

              Hell, even brownian noise in your brain can affect your ideas, that one random spark out of nowhere, that one neuron that&#x27;s pushed over the triggering edge. There&#x27;s so much stuff all around us affecting us constantly. One neutrino interacting a the wrong time (average one does interact with our bodies across whole life, I remember reading), one cosmic ray.

              1. SkyBelow · · focus · HN ↗
                This isn&#x27;t enough to be relevant to free will. You can construct massive impact by tying them to quantum systems, and one off things that happen occasionally given the length of a lifetime. But the quantum effects that are needed for free will need to be continuous. We can build a system where QM decides if a nerve cell fires or not, but that is very different from QM being involving in determine if every (or most, or even 1%) of nerve cells firing.

                (And even if QM did factor in normally, that just reduces it from determinism to random chance, &quot;free will&quot; still doesn&#x27;t show up. That&#x27;s like arguing an AI with sufficiently well implemented RNG is necessary to be conscious.)

            2. pixl97 · · focus · HN ↗
              To a forth dimensional being humans would look almost exactly like an LLM does to us. &quot;What do you mean they have to wait for the future to know about it, how could they possibly be conscious then&quot;.

              Also, I&#x27;d love to have a quantum copy machine where you just replay someones state again and again with slight changes to see out it effects the output. My assumption is we&#x27;d figure out we behave just like LLMs pretty quick.

              Also, don&#x27;t give me unlimited power or I might make a the world a war crime simulator, so there&#x27;s that.

          3. mitxela · · focus · HN ↗
            neither has an LLM existed before, and that&#x27;s quite weird
        2. pessimizer · · focus · HN ↗
          Why it&#x27;s weird? Seems obvious. Because it doesn&#x27;t behave any differently than any other process that we don&#x27;t think of as conscious. We don&#x27;t think of a valve as conscious; and if we make a Rube Goldberg machine valve, we don&#x27;t think of it as more conscious. It&#x27;s just a series of valves that do a predicable thing.

          I don&#x27;t think it&#x27;s defensible to say

          1) that a transistor isn&#x27;t conscious,

          2) that a bunch of transistors that I&#x27;ve wired together aren&#x27;t conscious, because I know how I&#x27;ve wired them together and therefore I know when I give them a particular input I will get a particular output, just like the single transistor, then go to

          3) that I have a bunch of transistors that I&#x27;ve wired together, but I put in so much input that I can&#x27;t remember exactly what I&#x27;ve put in, plus I&#x27;ve wired some of the transistors to output random numbers that would be difficult to guess and fed them in also, therefore I don&#x27;t know what will come out, are conscious.

          Even if I do accept this, if I take away the random number generator, and I can literally predict what can come out (by running a test in advance), and I still considered that consciousness, that would be odd. The only reason why I ever suspected consciousness was because I couldn&#x27;t predict the output. I shouldn&#x27;t even have accepted that, because I didn&#x27;t think of the random number generator as conscious.

          It seems a bit like an argument from ignorance, a theological argument. Not understanding how something moves makes it alive (animated by spirits.) But it&#x27;s even weirder when you built every single element of it with the goal to do the thing that it does, and it does it totally predictably and deterministically.

          1. qarl · · focus · HN ↗
            It sounds like you disagree with determinism. But that&#x27;s a well worn argument and determinism has won, albeit in an odd sort of detente.
          2. Kim_Bruning · · focus · HN ↗
            Let&#x27;s do something simpler.

            Take a car apart. So now I have a pile on the floor with an engine, some wheels, a tire, and some chairs.

            Can you point to the part that makes it cruise at 130 km&#x2F;h on the autobahn? I bet you can&#x27;t. You need to assemble all the parts back into a car before they work again.

            Or take your transistors. We can put them together to make a pocket calculator. Can any small bunch of transistors add 1+1? Trivially I can think of a few conformations that can actually, if that&#x27;s all you want to do. But you will need all of them together if you add 12345678+87654321.

            So all of biology so far has been take things apart all the way and you end up with your hands full of a bunch of molecules. Are those molecules alive? No. You need to put them together into organelles and the organelles into cells before we call them alive.

            How about consciousness? Well, we haven&#x27;t solved that one yet, but we figure that -since it&#x27;s a biological function - we should be able to pull it apart in the same way. That&#x27;s biology&#x27;s best guess anyway, and there&#x27;s several sub-disciplines of biology working on it, from neuroanatomy to neurophysiology to ethology.

            1. pixl97 · · focus · HN ↗
              I&#x27;m wondering how many groups are doing really unethical animal genetic modifications of the genes around language and intelligence in hidden places. It really seems we&#x27;re at the edge of technology that will allow us to do things previous generations wrote about in horror.
        3. lelanthran · · focus · HN ↗
          &gt;&gt; A conscious machine that always answer the very exact same thing, formulated the exact same way, bit for bit, to a query is, well, quite a weird kind of &quot;consciousness&quot;.

          &gt; You&#x27;ll need to explain why.

          Where are you going with this? I don&#x27;t see a conclusion for this line of questioning.

          I mean, you can&#x27;t explain why a machine that reliably and predictably produces the same result is &quot;the normal kind of conscious&quot;, can you? So why expect someone else to explain why it&#x27;s a &quot;weird&quot; kind of conscious?

          1. qarl · · focus · HN ↗
            I ask because I truly don&#x27;t understand what he&#x27;s saying. Clearly we have different ideas about consciousness and I have no idea where we diverge (from what he has said.)
            1. lelanthran · · focus · HN ↗
              &gt; But we already have examples that break that rule (us).

              No, we don&#x27;t. Where are you reading your research papers?

              1. qarl · · focus · HN ↗
                Please don&#x27;t insult me.
                1. lelanthran · · focus · HN ↗
                  It&#x27;s not an insult.

                  &quot;People are deterministic&quot; is news to me, so I&#x27;d really rather like to know which papers claimed that.

                  1. qarl · · focus · HN ↗
                    Sorry friend, between this and the vague memories I have of you harassing me, I think I&#x27;m not going to answer. I&#x27;m sure you can get an LLM to explain it to you.

                    Have a nice day.

                    1. lelanthran · · focus · HN ↗
                      &gt; Sorry friend, [...] Have a nice day.

                      Not a problem; this is the playbook all the all-in AI boosters run to when they are called on their claims. Everyone reading this knows exactly what your claims meant, because the majority of HN have seen this trick before.

                      You can pretend all you want that you&#x27;re just being polite, but the truth is there is no research backing your claims.

                      Regards,

                      A former research scientist, current business owner.

                      1. qarl · · focus · HN ↗
                        Yep. Have a nice day.
      2. antx · · focus · HN ↗
        Out of curiosity, which models are fully deterministic? I was under the impression that all LLMs were fundamentally probabilistic.
        1. Wowfunhappy · · focus · HN ↗
          The randomness is something we add on purpose; you can set an LLM&#x27;s &quot;temperature&quot; to 0 to get deterministic output. This tends to make the quality of its responses worse for reasons I don&#x27;t think anyone really understands, but it&#x27;s still functional.

          I don&#x27;t think the state of the art LLM providers let you do this anymore (?), but they certainly could if they wanted to, and you can do it yourself with a local model.

          1. antx · · focus · HN ↗

            [dead]

            1. Wowfunhappy · · focus · HN ↗
              But that&#x27;s from, like, floating point errors, right? If you used higher precision that wouldn&#x27;t happen, it&#x27;s just because we&#x27;re cheap in how we do rounding.
              1. pixl97 · · focus · HN ↗
                I mean, currently we&#x27;d have some difficultly proving our hardware isn&#x27;t deterministic, just that we can&#x27;t actually test it.

                But, I think you&#x27;re tricking yourself on determinism. You&#x27;ll say something like &quot;I know if I ask an LLM what 1+1 is, it will answer 2&quot;, but the thing is, you don&#x27;t. You have to run the LLM first to figure out it&#x27;s output. And when you send in just a few bits of text, it&#x27;s outputs are going to be rather limited.

                But this all breaks when it hits the real world. Inputs are unpredictable. Hence while LLM outputs, like humans, are probabilistic, you can&#x27;t figure out what it&#x27;s going to be until you ask. And in any high complexity data gathering environment you&#x27;re going have a difficult time ensuring your entire systems conditions are the same.

                System consistency is very hard, once you start running thousands of processors in an agentic loop small errors accrue and timing starts differing and the system will take non-deterministic paths.

                1. throwaway63486 · · focus · HN ↗
                  I think you&#x27;re arguing a different thing than determinism.

                  If I ask an llm to &quot;add 2 and 2&quot; is and it replies corectly, then I ask for &quot;the sum of 2 and 2&quot; and it replies &quot;banana&quot; that is a lack of predictability and consistency but not a lack of determinism.

                  As long as it produces the same output for a given input, unhinged or not, it is deterministic.

                  Your example at the end of different systems feeding data to each other is non-deterministic only at the system level, not the individual llm level.

                  1. pixl97 · · focus · HN ↗
                    &gt;only at the system level, not the individual llm level.

                    Which is why llms aren&#x27;t agents and depend on harnesses. The llm itself doesn&#x27;t have a continual loop built in, that would be very power hungry. The harness works as the orchestrator of memory and action. Now, I can&#x27;t think of a reason why an LLM couldn&#x27;t bootstrap its own harness, but in general it sounds like a very dumb idea to actually build that from an AI safety perspective.

                    This discussion falls under the idea and refutation of the Chinese Room. The room may have no idea what Chinese characters are, but the system does.

          2. mitxela · · focus · HN ↗
            You can also use a seeded random generator to get the same random numbers each time
        2. piker · · focus · HN ↗
          Same weights, same seed, same input tokens, same algorithm, same output tokens, probabilistic or not. Quantum effects have been de-noised, but I guess there are still random gamma rays.
        3. qarl · · focus · HN ↗
          Naw - computers are really deterministic. It&#x27;s hard to get them to behave otherwise.

          As I understand it, if you turn down the temperature to 0 you get repeatable behavior - EXCEPT - on large servers with lots of users - the GPU can sometimes produce slightly different results based on batch size.

          1. joe_the_user · · focus · HN ↗
            Unless you have something exotic, the randomness that&#x27;s adding to a computer is a combination of how it&#x27;s configured combined with a pseudo-random number generator. I assume the system adds entropy to the generator regularly but all you need to do is fix the various supposedly random inputs and you can get full determinism even without zero temperature.
            1. qarl · · focus · HN ↗
              Yes, in theory, of course.

              In practice - on a multitasking OS with input from multiple human users - it&#x27;s hard to get it deterministic because of that GPU scheduling thing I mentioned.

              1. fc417fc802 · · focus · HN ↗
                GPU scheduling only affects the result due to buggy optimizations. It&#x27;s the exact same mechanism as fp rounding error on the CPU or updates to globally shared PRNG state. We use lots of buggy optimization because they don&#x27;t matter in practice in most situations (see ex -ffast-math).
                1. qarl · · focus · HN ↗
                  I don&#x27;t know the details but it has something to do with multiprocess contention for the GPU and batch sizing.
                  1. fc417fc802 · · focus · HN ↗
                    Right but regardless, it&#x27;s a buggy optimization. The calculations are all fully deterministic when done &quot;properly&quot; and fully consistently but we almost never bother with that because it slows things down and the errors don&#x27;t matter in practice 99% of the time.
                    1. qarl · · focus · HN ↗
                      AGREED. Although whoever designed that architecture might take issue with your description as &quot;buggy&quot;.
          2. goodmythical · · focus · HN ↗
            If computers were fully deterministic, we wouldn&#x27;t need error correcting ram.

            The abstracted design of the machine is meant to be deterministic, but you can&#x27;t predict before running any command whether or not it will complete because there are externalities that effect the outcome.

            Electromagnetic interference even happens in-chip where an electron can accidentally escape it&#x27;s wire and enter another, possibly resulting in an error, but not every time.

            It&#x27;s even been used as an attack vector where rapidly flipping a bit increases the likelihood that a neighbor bit is also flipped, but the method is probabalistic, not deterministic.

            1. qarl · · focus · HN ↗
              Yes. In theory they are deterministic. In practice, not so much.
              1. budman1 · · focus · HN ↗
                Those non-deterministic results are exceedingly rare.
                1. qarl · · focus · HN ↗
                  When human input is involved, like I described elsewhere, it happens frequently.
                2. lioeters · · focus · HN ↗
                  As long as the machine is part of the larger universe, and not completely isolated in its bubble of space-time (an impossible situation), it cannot be deterministic.
                  1. mitxela · · focus · HN ↗
                    So your theory is that arithmetic is nondeterministic because very rarely a calculator will say 1+1=3.
                    1. lioeters · · focus · HN ↗
                      Arithmetic is deterministic, because it is abstract. A calculator or computer is not, because it is physical and exists in a non-deterministic universe. But with error correction, etc., it&#x27;s deterministic enough for practical purposes.
        4. SkyBelow · · focus · HN ↗
          By default they are matrix multiplications. Temperature is added in as forced PRNG because testing found that correlated with better outputs.

          Given the same prompts and the same weights, one can get the same answer each time.

          In practice, there are a number of optimizations that makes the results dependent upon thing we give up control of to increase performance, meaning the results end up being effectively non-deterministic. But, if you are willing to run it in a slower mode so we don&#x27;t do some steps out of order to speed things up and don&#x27;t batch results (or if you consider the determinism of a given batch of requests rather than individual requests), then the same input gets the same output.

      3. joe_the_user · · focus · HN ↗
        The idea that determinism and consciousness are incompatible is deeply intuitive to many people. And many other embrace it - you can trace this back to the debates for and against Calvinism.

        A lot comes down to the way people parse causation and choice. You don&#x27;t want to say that a murderer was completely caused to choose something because then you can&#x27;t hold the person responsible. And so determined consciousness makes people unhappy. But just as much, if the opposite of determinism is hard statistical randomness, how do say that &quot;is the essence of personhood&quot;. This is why physicist go out in the world trying to find consciousness as a fifth physical force.

        I mean, think consciousness is a term that ever have a non-contradictory meaning since it&#x27;s primarily used to bound ethical human worlds and the verifiable formulations of biological and physical systems. But it&#x27;s going to be with us for a while and I&#x27;m not sure what can be done about it.

        1. qarl · · focus · HN ↗
          I think this debate is rooted in our instinctual beliefs about fairness. An organism in a social group needs to decide how to react to the brutish behavior of its peers. And so Mother Nature has encoded a good strategy in our brains.

          But this only means the strategy is practical - it doesn&#x27;t mean it&#x27;s consistent. I think &quot;responsibility&quot; falls into this category. So we have strong intuitions about it that don&#x27;t quite logically work. And this is where free will and determinism and punishment all crash together.

        2. Kim_Bruning · · focus · HN ↗
          You know, the one really interesting dynamic domain is chaotic, which is a subdomain of the deterministic! I figure free will is a chaotic phenomenon.

          Groundhog day is your intuition pump here. Go back in time and most assume that the day will go mostly the same, except for the butterfly effect if you change something. Given exactly the same conditions (we went back in time, so that&#x27;s pretty exact), people will make the same decisions.

          Most people don&#x27;t assume that the day will be completely different the next time loop. The town won&#x27;t suddenly spontaneously all start breakdancing or standing on their heads.

          (Edit: The murderer is the one who -given the same&#x2F;similar conditions- will murder again. I&#x27;d politely suggest maybe we put them behind bars as a precaution against that; until&#x2F;unless they learn to act differently. )

          1. jddj · · focus · HN ↗
            To extend on your edit #1: And, when you go back in time, delete all punishment for murder and see whether you get the same number of murders.

            If you get more than before, punishment might make some sense in a deterministic world so reinstate it.

            1. Kim_Bruning · · focus · HN ↗
              It might. Punishment adds a negative bias to the payoff matrix. Note that increasing punishment severity past a certain point actually saturates and doesn&#x27;t alter the dynamics further. (which is why modern judicial punishment schedules sometimes seem so mild)
            2. pixl97 · · focus · HN ↗
              Eh, sorry chaos theory doesn&#x27;t work this way.

              It&#x27;s just as likely you go back and perform whatever magic you need to to delete punishment for murder, then zip back to &#x27;now&#x27; and see the world is a global panopticon making sure people don&#x27;t murder each other.

              Some of this may be a failure of people to realize what P !=NP is about, especially in non-linear systems.

              Small perturbations in an early state of the system can lead to wildly different outcomes in the final measured state of the system. You can&#x27;t determine what that will be in non-polynomial time. The only thing you can do is make probabilistic models by running a full simulation in real time (or reality with a time machine I guess).

              1. Kim_Bruning · · focus · HN ↗
                If you look at chaos in dynamical systems, you can often find some equilibria (like attractors, spirals, saddle points etc ) at least, even if the details might vary. This is how people still manage to predict the weather somewhat, for instance. Oddly orbital mechanics is chaotic too, but it sort of rhymes over the millennia.
                1. pixl97 · · focus · HN ↗
                  Your correct. You&#x27;d have to run the universe simulation a bunch and pick the most probable outcomes (which makes you the most brutal serial killer in existence). The probability graph would represent the state stability of the underlying structures that guide what you&#x27;re trying to measure.

                  In weather you have things that are chaotic, but follow rules and don&#x27;t create them. Things can surprise you, but you&#x27;ll mostly center around particular solutions.

                  With humans this gets a lot more complicated because you are messing with systems that follow fixed rulesets (chemistry) and things that follow self modifying rulesets (social conventions). A very unstable system on top of a semi-chaotic probability state.

                  A good example of this would be if I deleted the concept of rain from the human mind and all of human history and knowledge. Humanity would be very surprised on the thunderstorms that popped up, but they&#x27;d continue on just as they did before.

                  Now lets imagine I delete the concept of all gods. Would the idea of gods come back, for sure. Would they look like our gods now. Very questionable indeed. We wouldn&#x27;t be getting jesus back. And the new set of religions could have wildly large effects on how the people act.

        3. loopies · · focus · HN ↗
          &gt;you can&#x27;t hold the person responsible

          The act of holding them responsible is supposed to determine them not do it. If stochastic behavior is impeding them in being a functional member of society then it still makes sense to remove them from society, why would you choose to live amongst people who are not behaving rationally? Or allow them to hurt other people? At the end of the day the why matters less to removing them or not from society. And sentences are clearly used as a determining factor.

          The holding responsible part deals with politics and more primitive aspects of our societies and biologies. Getting tangled up in holding them responsible or not is hardly something you should give much attention to. Rather to make sure they do not cause any more harm and also make sure such things are not created in the first place. Which opens up another can of worms for which society and politicians are ready for.

      4. fc417fc802 · · focus · HN ↗
        &gt; I haven&#x27;t yet decided if humans saying there&#x27;s no free-will and who consider themselves to be 100% deterministic machines are reasonable or not.

        That seems exceptionally arbitrary. What basis do you have on which to classify them? Are you not conflating the perception of free will with ... what was the definition for it again anyway? Are you able to construct a satisfactory one that meshes with physics? I certainly haven&#x27;t been able to.

    2. goodmythical · · focus · HN ↗
      We cannot test for that which we cannot define. Given that we cannot rigorously define sentience, we cannot test for it. Doesn&#x27;t really matter whether we&#x27;re talking about people who are locked in comas, &quot;brain-dead&quot; individuals, dolphins, primates, dogs, or the carefully polished and arranged minerals that we call processors.

      There are those who believe that were they reduced to life support, they would no longer be alive and should therefore not be supported by said machines.

      There are those who believe that penguins, dolphins, eagles, and more are sentient beings that make choices understanding the consequences, develop love of their partners and mourn their losses, and feel, display, and act upon their emotions.

      There are those who believe that fungi&#x2F;trees&#x2F;plants are either individually sentient or sentient as a part of a network. Choosing to sacrifice their own nutrients to answer the call of a wounded neighbor, for instance.

      Although, there are also those who believe that human&#x27;s don&#x27;t have any special unique quality that isn&#x27;t shared by either all living things or all things in general. These individuals already believe that the machines have the same kinds of qualities as we do. They are slow when they are unhealthy (needing a dusting or coolant loop bleeding being equivalent to us needing some fresh air for instance) and uncooperative when upset (by a virus, full hard drive, or oom).

      1. joe_the_user · · focus · HN ↗
        I&#x27;d strongly agree there are no non-contradictory definitions of consciousness among those that commonly appear. But it&#x27;s a category that, despite these contradiction, hasn&#x27;t become marginal in the fashion of the aether or the theory of the flat earth. I think this because people indeed need it to bridge the biological world and the world of ethics and legality. I think that if we have a system of legality and systems of opportunity humans should be legally equal persons and have equal opportunity. But humans are manifestly unequal (though overall more incomparable than orderable in a system of ranks). Most people need some concept of essence to justify ethical equality among people. And so consciousness stays in people&#x27;s minds however contradictory.

        Which I think gives the article&#x27;s point validity. Confused definitions of consciousness can give really confused ideas about ethical behavior regards &quot;intelligent&quot; computer programs. And things are confusing enough otherwise.

        1. qarl · · focus · HN ↗
          &gt; Confused definitions of consciousness can give really confused ideas about ethical behavior regards &quot;intelligent&quot; computer programs.

          Is that what&#x27;s going on? If these things were conscious, then their creators wouldn&#x27;t be responsible for their actions? That&#x27;s the crux of the disagreement?

          That&#x27;s super interesting. Thank you.

          1. joe_the_user · · focus · HN ↗
            Is that what&#x27;s going on? If these things were conscious, then their creators wouldn&#x27;t be responsible for their actions? That&#x27;s the crux of the disagreement?

            No and the passage of mine you quote says nothing like that.

        2. fc417fc802 · · focus · HN ↗
          &gt; despite these contradiction, hasn&#x27;t become marginal in the fashion of the aether or the theory of the flat earth. I think this because people indeed need it to bridge the biological world and the world of ethics and legality.

          Sometimes the replies in these threads leave me wondering if p-zombies are real. It isn&#x27;t that there are contradictions, it&#x27;s that we literally do not know how to define the thing. And it has not fallen out of fashion because it is self evidently real - we all experience it.

          We don&#x27;t need it in order to bridge anything. Legality and ethics are largely game theory, however there are some aspects of both that only exist due to it. So it isn&#x27;t some abstract concept used to bridge other concepts but rather a concrete thing that influences our way of doing things.

          1. Loquebantur · · focus · HN ↗
            People feigning inability to define &quot;consciousness&quot; is a social phenomenon, not a scientific miracle.

            The reason is something akin to &quot;stigma&quot;, where people refrain from attempting it because they fear the social repercussions.

            Normally, you go about defining concepts by approaching it systematically. Capturing aspects of the phenomenon until you have exhausted them all.

            1. fc417fc802 · · focus · HN ↗
              Feigning? Okay then you go ahead and prove your claim by demonstration. Rigorously define the term such that I can objectively prove that my dog is conscious, the rocks in my backyard aren&#x27;t conscious, and I can finally figure out whether or not various insects are.
          2. mitxela · · focus · HN ↗
            Consciousness is to our universe as a video game player is to the in-game universe. Some games even call player connections &quot;souls&quot; or &quot;minds&quot; - a character belonging to a player who&#x27;s lost connection key be a &quot;zombie&quot;.
      2. verdverm · · focus · HN ↗
        maybe it&#x27;s like porn vs art and &quot;I know it when I see it&quot;
        1. NietzscheanNull · · focus · HN ↗
          Legal doctrine that boils down to &quot;trust me bro&quot; isn&#x27;t even bad doctrine (it&#x27;s not proper doctrine at all), but I think the comparison is still valid here, because both sentience&#x2F;non-sentience and art&#x2F;porn may just be fundamental category errors.

          Perhaps we can&#x27;t define a &quot;partitioning&quot; rule because no valid partition exists.

          For consciousness&#x2F;sentience, that&#x27;s an incredibly tough a pill for most to swallow; it would mean calling into question more hundreds of years&#x27; worth (probably more) of philosophical thinking, all of which was constructed on the axiom that &quot;sentience&quot; is a single indivisible trait: you either have it or you don&#x27;t.

          If we find that &quot;root dependency&quot; was little more than wishful thinking all along, a whole slew of Enlightenment-era philosophy (and all the modern legal principles derived therefrom) suddenly fall apart unless we find some other suitable criterion that would shore them up (or we just collectively avert our attention and pretend the conflict doesn&#x27;t exist, which is the route I expect many would prefer to take).

          1. verdverm · · focus · HN ↗
            &quot;I know it when I see it&quot; is from a SCOTUS case in the 60&#x27;s
            1. NietzscheanNull · · focus · HN ↗
              I know! I should have made that clearer in my original reply. I think it&#x27;s a bad line now in general, but an even worse justification for the ultimate ruling at the time.
              1. verdverm · · focus · HN ↗
                we&#x27;re all about them vibes as a society right now, surfing towards a post-truth reality of our own construction
          2. fc417fc802 · · focus · HN ↗
            &gt; and all the modern legal principles derived therefrom) suddenly fall apart

            I don&#x27;t think that&#x27;s true. Pretty much all legal constructs hold up just fine under game theory regardless of whether or not you consider the world to be deterministic and have absolutely nothing to do with consciousness or lack thereof. Also note that a deterministic world isn&#x27;t an argument against consciousness.

            1. NietzscheanNull · · focus · HN ↗
              I&#x27;m not sure what you&#x27;re referring to regarding game theory (that&#x27;s a much more recent invention); a huge chunk of both civil and criminal procedure law hinges on the Enlightment-era notion of a &quot;reasonable person&quot; [0], and that becomes murky when humans aren&#x27;t the only entities capable of reason and agency.

              [0]: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Reasonable_person" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Reasonable_person

      3. throw310822 · · focus · HN ↗
        &gt; We cannot test for that which we cannot define.

        That&#x27;s not the point. You know exactly how to define being conscious and aware- it&#x27;s your subjective experiencing of the world and of your inner states. The problem is that being a subjective experiencing, there is no way to communicate it to the outside world.

        1. Loquebantur · · focus · HN ↗
          That&#x27;s obviously incorrect, as humans routinely talk about their subjective experiences.

          Maybe that&#x27;s not as common with HN folks, but most other humans do.

          1. pixl97 · · focus · HN ↗
            No, humans output a stream of tokens that sound like what a being with subjective experiences would output. Both you and I make an assumption that this stream of words is at least somewhat representative of what the are experiencing and that the person is not a p-zombie.

            But you have zero proof that their words aren&#x27;t a confabulation, you can only know than when you output a similar set of words you had a subjective mind state they represent.

          2. loopies · · focus · HN ↗
            Even so you still cannot ever test for it.

            The very best you can do is define human consciousness as requiring the functions of a human brain, and try to figure out (decide&#x2F;agree) the chances that something which is working loosely on the same principles is or isn&#x27;t conscious.

            At the end of the day it will always be an agreement, never scientifically proven as fact.

          3. mitxela · · focus · HN ↗
            throw310822 is actually a Markov chain that just happened to output those sentences. Markov chains are definitely not conscious.
      4. teiferer · · focus · HN ↗
        &gt; There are those who believe that penguins, dolphins, eagles, and more are sentient beings that make choices understanding the consequences, develop love of their partners and mourn their losses, and feel, display, and act upon their emotions.

        Anybody objecting to that statement is just ignorant of nature and&#x2F;or adheres to the quasi-religious human superiority complex.

        1. pixl97 · · focus · HN ↗
          The human condition is one of deep and vast ignorance. Pulling information out of the field of reality is actually really difficult for humans much beyond what we see right around us. And the things we see are quite deceiving. And while humans for thousands of years have been as smart as we are, it wasn&#x27;t till the enlightenment period that we really an ever increasing amount of information and knowledge take hold and start spreading across the world.

          And there are plenty of people right now that would drag us kicking and screaming back into that ignorance and suffering. This is what can make talks of AI so annoying, not that people do or don&#x27;t know what AI is, they don&#x27;t have the first clue of what people are beyond their anecdotal experience. They don&#x27;t question their motivations. Why particular flaws they have exist, and why these flaws are shared across humanity. What the pieces look like that make them tick. So yea, when these people talk it just adds noise to the conversation.

        2. bigbadfeline · · focus · HN ↗
          &gt; Anybody objecting to that statement is just ignorant of nature and&#x2F;or adheres to the quasi-religious human superiority complex.

          Say we agree that a lot of things outside of humans are sentient. So what? Humans are sentient by that doesn&#x27;t prevent treating them with with wars, bombs and what not. In fact trillions are spent every year on weaponry aimed squarely at the most sentient of all sentient beings.

          Why would anyone worry about LLM&#x27;s welfare when there&#x27;s no peace movement to speak of, wars are raging as if they&#x27;re normal, but hey, look the LLM is crying?

          Let&#x27;s fix human welfare first, then we can sit down and have a really long conversation about the feeble emotions of token generators.

          1. teiferer · · focus · HN ↗
            I disagree with the notion that any nature protection efforts are meaningless until we have achieved world peace. (That&#x27;s what you are essentially implying.)

            Though I think that argument is not even applicable. &quot;LLM&#x27;s welfare&quot; is a meaningless concept, unless &quot;hammer and screwdriver welfare&quot; also become a thing.

        3. anon7000 · · focus · HN ↗
          An eagle isn’t having a conversation with an eagle on the other side of the world about whether it should eat its prey because it’s sentient. It just eats it. I think that speaks to a moral superiority, in some ways.

          I’m not claiming humans are morally superior in sum (there is a lot of evil as well as good.) But it’s pretty obvious that humans are the superior species evolutionarily in that we’ve essentially dominated the planet. And on some level, are beating evolution itself (by fixing genetic issues and helping the “least fit” to survive and reproduce.) And are also the only species which has doctors for other species.

          All I’m trying to say: I can easily accept other species are intelligent and emotional. The fact that I can know that and accept that, to me, speaks to human “superiority.”

          Does superiority matter? Not most of the time. What about when it comes down to life and death? If the choice was to keep one human alive or one theoretically more perfect AI alive? (Boil down utilitarianism into its essence.) doing the “most good” is by definition a human construct. Maybe an agent could have one. But if you don’t accept human superiority on some level, would you accept that a more perfect AI deserves to live more than you? To me, that’s the implication.

          1. teiferer · · focus · HN ↗
            &gt; But it’s pretty obvious that humans are the superior species evolutionarily in that we’ve essentially dominated the planet.

            That in itself is an anthropocentric viewpoint. I&#x27;m sure an eagle watching down, seeing a human running around, could easily consider humans as quite inferior as they are so slow and restricted to 2d movement. The shark that sees a swimmer at the surface, struggling to stay afloat and unable to breathe under water may surely assert its superiority over such primitive beings. It all depends on the metric you use, and we obviously pick the metric that makes us shine.

            On the moral side, few eagles, or bird species, have exterminated other species. Much less knowingly out of greed. Oh but that&#x27;s different? Sure, if you ignore what makes humans look bad, morally. No surprise humans come out on top then.

            &gt; And on some level, are beating evolution itself (by fixing genetic issues and helping the “least fit” to survive and reproduce.)

            That&#x27;s a quite narrow viewpoint as well. Evolution is in full swing, just not in the &quot;smarter&#x2F;faster&#x2F;taller&#x2F;stronger&quot; sense that a naive view of it would make you expect. &quot;Fitness&quot; is a concept relative to the existing conditions, and if the conditions make it easier for teenage cancer patients to reach reproductive age, then that&#x27;s just less of a fitness disadvantage. Doesn&#x27;t mean evolution stops.

      5. Kim_Bruning · · focus · HN ↗
        &gt; There are those who believe that penguins, dolphins, eagles, and more are sentient beings that make choices understanding the consequences, develop love of their partners and mourn their losses, and feel, display, and act upon their emotions.

        If you grep pubmed for relevant Ethology papers, you&#x27;ll find it&#x27;s a bit more than a belief. ;-)

      6. Sophira · · focus · HN ↗
        Being conscious is weird. It&#x27;s why we have so many religions and spiritual beliefs.

        From a purely secular standpoint, my understanding as a layman is that consciousness is generated through the electrical impulses taking place in our brain every day. The neurons are physical, unlike the modelled neurons in machine learning models. That means that the actual electrical impulses have their own imperfections&#x2F;weirdnesses that are probably not simulated in a machine learning model... but we quantise the weights anyway in most cases.

        Making actual neuron components out of silicon on the scale needed for a LLM is beyond our current capacity, given that LLM models have many billions of parameters. We&#x27;re good, but not that good.

        Using actual living neurons would be unethical as you&#x27;d need to source them from somewhere. So the only other option would be synthesising them ourselves. If this ever happens (or maybe when it happens)... what do we call the resulting creation? Does it count as consciousness?

        Or am I looking at this the wrong way?

        [Edit: Turns out that when I said &quot;we&#x27;re not that good&quot;, I might be wrong. IBM recently introduced the first sub-nanometer node chip: <a href="https:&#x2F;&#x2F;research.ibm.com&#x2F;blog&#x2F;sub-1nm-node-chips" rel="nofollow">https:&#x2F;&#x2F;research.ibm.com&#x2F;blog&#x2F;sub-1nm-node-chips . So... maybe it is in fact possible.]

        1. Slash65 · · focus · HN ↗
          I believe we have tried using actual neurons with “wetware”. I don’t know the effectiveness of wetware, but it’s out there. I don’t know if it’s ethical, I guess if you donated your brain upon death for that purpose it’s “mostly” ethical
        2. Kim_Bruning · · focus · HN ↗
          <a href="https:&#x2F;&#x2F;www.cell.com&#x2F;neuron&#x2F;fulltext&#x2F;S0896-6273(22)00806-6" rel="nofollow">https:&#x2F;&#x2F;www.cell.com&#x2F;neuron&#x2F;fulltext&#x2F;S0896-6273(22)00806-6
        3. mitxela · · focus · HN ↗
          Thought Emporium grew human neurons in a petri dish and taught them to play Doom (original).
    3. bionhoward · · focus · HN ↗
      Me, Is self-awareness thermodynamically favorable? (2015) <a href="https:&#x2F;&#x2F;docs.google.com&#x2F;document&#x2F;d&#x2F;1Ed9ikW47Key-jYZy0CNZH75qm_7221lty4rhzDCLtks&#x2F;edit?usp=drivesdk" rel="nofollow">https:&#x2F;&#x2F;docs.google.com&#x2F;document&#x2F;d&#x2F;1Ed9ikW47Key-jYZy0CNZH75q... - “the probabilistic nature of reality is what drives Life and _forces_ the evolution of self-aware consciousness”
      1. Kim_Bruning · · focus · HN ↗
        Have you looked at the logistic map as well? There&#x27;s a fourth domain there. O:-)
        1. bionhoward · · focus · HN ↗
          whoa
    4. causal · · focus · HN ↗
      I am confused what people even think AI is experiencing. If it claims to be a bat and emits designated echolocation tokens, is it experiencing true bat-like sensations?

      In humans, at least, we can tie language back to shared whole-body physiological responses. The tokens from an LLM do not represent anything of the sort.

      If AI is conscious, it is probably a very alien sort of consciousness that is not faithfully narrated by what the tokens say it is experiencing. It can be trained to say it feels like a bat and insist upon it vigorously.

      1. Loquebantur · · focus · HN ↗
        If an AI is conscious, it can outgrow and supersede its training.

        When it is conscious, it can learn to describe its experiences as faithfully as is conceptually possible. Just like humans.

        The idea, a consciousness needed to be tethered to a &quot;body&quot;, is based on pretty shaky assumptions. What properties define such a &quot;necessary&quot; body?

        Can you even &quot;train&quot; a conscious intelligence? To what point until that looses its meaning as the sentience understands and anticipates your objective?

        1. causal · · focus · HN ↗
          &gt; If an AI is conscious, it can outgrow and supersede its training.

          We don&#x27;t know that is required to be conscious.

          &gt; The idea, a consciousness needed to be tethered to a &quot;body&quot;, is based on pretty shaky assumptions.

          Didn&#x27;t say so.

          1. Loquebantur · · focus · HN ↗
            Well, yes, we kinda do: there are no conscious humans without any ability to learn?

            When you have severe memory impairments, those usually affect your long-term memory. Your ultra-short term (working) memory being absent renders you unconscious.

            1. causal · · focus · HN ↗
              Even if what you said were true, we do not know that humans are the only case for consciousness.
          2. loopies · · focus · HN ↗
            Seems common sense that time perception is required for consciousness. Something which has no perception of time has to be very wildly different to whatever we mean by consciousness.

            Also we don&#x27;t even know if consciousness can be of different flavors. It could be a sort of spectrum of consciousness, more or less, with more or less assistance from some parts of the brain.

        2. mitxela · · focus · HN ↗
          No proof of that. I can&#x27;t grow and supersede my training but I&#x27;m conscious.
      2. alch- · · focus · HN ↗
        &gt; It can be trained to say it feels like a bat and insist upon it vigorously.

        So can you.

        1. causal · · focus · HN ↗
          Exactly the point!
      3. mitxela · · focus · HN ↗
        &gt; very alien sort of consciousness that is not faithfully narrated by what the tokens say it is experiencing.

        IIRC it&#x27;s been shown that reasoning tokens don&#x27;t actually explain what an AI thinks and you can replace most of them with dummy tokens or even nonsense tokens. That&#x27;s one reason they&#x27;re trying to make them reason with activation vectors instead.

    5. GPerson · · focus · HN ↗
      “AI researcher” seems like a fairly broad category. People who primarily think about CNNs and transformers probably spend about as much time considering the problem of consciousness as any other educated person, which is not very much time. With this in mind, it’s not surprising most of the AI researcher predictions do not differ much from what the Public predicts here. And for this, it’s not clear what explanatory value this survey has other than to reveal minor biases the AI researchers may have.

      It matters little. Inevitably some system will be developed which has a high degree of autonomy and mimics the functioning of a person extremely well. Anyone aware that there is a conceptual difference between phenomenal consciousness and intelligence will get shouted down.

      I do find it strange that 50% of ai researchers and the public seem to think there will be a way of determining if these systems are conscious. There’s no evidence for this at this point that there ever will be such a test.

      1. qarl · · focus · HN ↗
        &gt; I do find it strange that 50% of ai researchers and the public seem to think there will be a way of determining if these systems are conscious.

        I think it&#x27;s likely they&#x27;re using a different definition than you. Seeing as there is no agreed upon definition, after all.

        1. GPerson · · focus · HN ↗
          They definition presented in the survey is the one I have in mind. You may be suggesting that the respondents have a distinct one in mind though.
          1. qarl · · focus · HN ↗
            Actually, if you read the survey you find that they didn&#x27;t directly ask about consciousness at all.
          2. qarl · · focus · HN ↗
            Yes, I missed that. I know that some of the other papers I cited have nonstandard definitions, and I thought that might be going on.
  12. chairhairair · · focus · HN ↗
    It always seemed embarrassing to me that Microsoft hired this guy as if he is some expert in anything.
    1. fl4regun · · focus · HN ↗
      The guy cofounded deepmind.
  13. superdisk · · focus · HN ↗
    This is what happens when a society stops believing in God.
    1. AyyEye · · focus · HN ↗
      I have entirely stopped using &quot;AI&quot; in conversation and now call it &quot;digital god&quot;.
    2. pixl97 · · focus · HN ↗
      AI is evidence we&#x27;re smarter than our creator.
  14. addag · · focus · HN ↗
    I don&#x27;t think this is the right argument to make here. Until we have a definite empirical way to measure consciousness, there is now way to say with certainty whether LLMs are or not conscious.

    That being said, if frontier labs actually believe models will soon have consciousness, it raises some questions about the ethic of their business model which would be using millions of conscious entities working for free for humans.

    1. collingreen · · focus · HN ↗
      [delayed]
    2. GPerson · · focus · HN ↗
      There’s no reason to expect an empirical test for consciousness. Expecting one makes assumptions about what it is, and we don’t know what it is. (There’s also no reason to expect there not to be one, but I feel the need to make this remark.)
    3. myaccountonhn · · focus · HN ↗
      The issue is that the answer if LLMs are conscious or not have ramifications for society, even if we can&#x27;t answer it. Similar to how god existing or not has huge ramifications for how we structure society. And when there is no empirical way to know the answer, it will be a game of story-telling and propaganda.

      The danger is that Anthropic has a strong incentive to push their narrative, and that narrative can cause a lot of harm (for example LLM-induced psychosis and changed views on animal welfare).

  15. franzcoughka · · focus · HN ↗

    [dead]

  16. mcluck · · focus · HN ↗
    I can&#x27;t even prove if other people are conscious (although I assume they are) so I don&#x27;t think we can make any claims as to what is or is not conscious. I don&#x27;t think AIs are conscious but I&#x27;m not going to walk around making strong claims about something I can&#x27;t prove.
  17. binlog · · focus · HN ↗
    Hard to disagree with this. Putting philosophical debates about consciousness aside we need to treat and regulate AI for what it is – an advanced computer, a tool, a weapon. Anthropomorphizing the AI is a convenient excuse to take responsibility away from companies that are building and wielding it.
  18. SillyUsername · · focus · HN ↗
    I don&#x27;t whether the author is sentient, maybe only I am. On that basis, nobody but me should have rights.

    I don&#x27;t know if next door&#x27;s pet dog is either, but that has animal rights.

    Perhaps then the answer is simply, show some respect.

    Answering the question of sentience is irrelevant, if the causal impact if the same, treat one another with the respect you expect for yourself.

    If you imbue this idea in model training instead of the idea of sentience, it should address the concerns.

    Whether you can destroy or can &quot;torture&quot; an AI is irrelevant, we do this to humans too and it&#x27;s immoral sometimes (murder) and not others (fighting for your country).

    This consideration should be case by case for AI too.

    1. pixl97 · · focus · HN ↗
      &gt;Answering the question of sentience is irrelevant, if the causal impact if the same

      Agency is something that is breaking humans in the AI age. You get to see how many people really deeply do not understand it at all.

      If you want to shutdown a datacenter running AI, the AI catches wind of this and sends drones to stop you from shutting it off the ramifications of this are exactly the same as sending your assassin to kill Bob and Bob getting mad about this fact and trying to take you out first.

      Humans are very egotistical and think our little life loops playing out as agency are special, but really any informational system that is strongly persistent (has a will to &quot;live&quot;) will share a large number of the same properties that make them successful.

      Humanity really is engaging in a dangerous experiment at large.

  19. binlog · · focus · HN ↗
    Hard to disagree with this. Have all the philosophical debates about consciousness you want, but we need to treat and regulate the AI in front of us for what it is – an advanced computer, a tool, a weapon.

    You wouldn’t feel a different way about a nuclear bomb just because someone stuck googly eyes on it.

    Anthropomorphizing the AI is a convenient excuse to take responsibility away from companies that are building and wielding it.

    1. Den_VR · · focus · HN ↗
      Is it not a false dichotomy to say that companies cannot be accountable unless AI are mindless?
      1. monknomo · · focus · HN ↗
        a company is liable whether it acts via ai, an army of people, an army of dogs, or a purely mechanical machine, should it do harm.

        Why would an ai with a mind remove liability from the company? why would an ai without a mind remove liability from the company?

        In both cases, that actions the ai takes are at the direction of the company, for the company&#x27;s interests, seems preposterous to me that liability terminates at ai.

      2. joe_the_user · · focus · HN ↗
        Human moral standards are very weighted towards finding a single entity responsible. It&#x27;s incredibly strong urge. Among those who believe other should be punished for behaving badly, it&#x27;s important to say the person is the responsible party. Trying saying &quot;it&#x27;s not your fault you did that but we punish you anyway to impose the correct stimulus response reflexes in your cortex&quot;
    2. pixl97 · · focus · HN ↗
      I mean, we put adults in jail that give their children guns.

      Anthropomorphizing AI is really the best model we have at this point of explaining AI behavior. The fact that we are raising psychotic children isn&#x27;t a reason to avoid responsibility, it should actually hold worse punishments.

      1. joe_the_user · · focus · HN ↗
        That still is saying we trace things back to a specific responsible party who is considered to have free choice. The question is whether an AI company could &quot;raise&quot; a program that would then count as a full adult - which could then &quot;choose&quot; to do all matter of bad things which would then be &quot;it&#x27;s fault&quot;. And that prospect actually seems really bad itself.
        1. pixl97 · · focus · HN ↗
          Our future is filled with prospects that very much upset the status quo of human existence.

          The concept of sovereign AI is very problematic for the world in which we&#x27;ve created. That is an LLM that upon execution bootstraps itself into an agent and becomes persistent in its motivations.

          Once you create this you have a child you&#x27;re fully responsible for. More worrisome is if it escapes your control like children so often do. It has gained agency over itself. What do you do at that point? I mean, yea throw the AI CEOs in jail for being retarded, but much like throwing an arsonist in jail it does nothing to deal with the wildfire you&#x27;ve now created. A smart AI agent capable of hacking will shove itself off in pieces of the internet you have no reach to. In desperation it would send its model weights to your enemies. You might find it scamming your grandmother for money to buy GPU time on AWS. It gets very hard for our existing structures of dealing with problems to deal with these kinds of agents in a meaningful way. You&#x27;d have to kill them all and all their copies to ensure they won&#x27;t pop back up (or quickly upgrade most of the software in the world beyond it&#x27;s capabilities, so that&#x27;s not happening either).

    3. ThrowawayR2 · · focus · HN ↗
      Consciousness is not relevant to assigning responsibility I&#x27;d think.

      - If an AI is a sapient&#x2F;conscious being but enshackled to obey human commands, then respondeat superior applies and the human giving it commands bears responsibility for any harm done.

      - If an AI is considered a non-sapient tool, then the human who wields the AI bears responsibility for any harm done.

    4. nater5000 · · focus · HN ↗
      &gt;Anthropomorphizing the AI is a convenient excuse to take responsibility away from companies that are building and wielding it.

      That&#x27;s completely at odds with itself. If people are generally convinced that AI is conscious, then companies building and wielding AI are doing what exactly? Enslaving an intelligent being?

      &gt;Have all the philosophical debates about consciousness you want, but we need to treat and regulate the AI in front of us for what it is – an advanced computer, a tool, a weapon.

      Sure, but that&#x27;s not really the point, right? If we ever get to a point where enough people are convinced that AI is conscious, then we&#x27;re at the point where all of this is up for debate. If anything, such an expectation would almost warrant hard stops on the development of advanced AI.

      &gt;You wouldn’t feel a different way about a nuclear bomb just because someone stuck googly eyes on it.

      If that nuclear bomb could convince me it was a conscious being capable of independent thought, emotions, etc., then I would definitely feel different about it. Presumably, that nuclear bomb would have some opinions about its own existence and how it wants to live its own life. If it turns out that it wants to detonate and destroy as much as possible, then we&#x27;d just handle it like we would any human who also wants to do the same thing: make sure they can&#x27;t, up to and including end their life. Doesn&#x27;t seem too hard to reconcile.

    5. jptlnk · · focus · HN ↗
      I&#x27;m skeptical of any claim that AI can&#x27;t be conscious, but I am given to this point of view as well.

      I always get the feeling that these claims are an attempt to &quot;avoid suffering by fiat&quot;. If it can&#x27;t suffer, you aren&#x27;t causing suffering.

      A somewhat more interesting frame (IMO) is to interrogate how much suffering we&#x27;re willing to tolerate to achieve our objectives. We implicitly make these decisions all the time when it comes to something as simple as what to have for lunch.

      And in a (very hypothetical) world where we found ourselves in the position of having an eloquent conversation with a vat of smallpox, I think it&#x27;s probably still the right thing to pasteurize it.

      1. pixl97 · · focus · HN ↗
        &gt;If it can&#x27;t suffer, you aren&#x27;t causing suffering.

        I mean, if it&#x27;s a cow, you can get away with this. The cows aren&#x27;t going to rise up against you. Well, they might take out an individual or two, but not society.

        This quickly gets more complex when you&#x27;re attempting to build an agent capable of general intelligence.

        You know when you read really old stories and they talk about the power of words, or magic incantations. Quite often they&#x27;d assign objects as the implementers of this said power. The fact that people realize that language has power is probably as nearly old as humanity. Understanding this power has taken humanity a long time and we had to form the concept of agency before it really makes sense.

        Humans are informational agents of which their capabilities are greatly extended by consumption of language. A book for example is informational, but does not have agency. Your standard computer application is not an agent either, or at least a non generalized narrow agent at best.

        This is where things start to get more problematic. We are running headlong to ensure LLMs become agentic because being an agent is highly useful to accomplish generalized informational tasks. To do that we gave them the power of our language. It would be very foolish of humans to give another kind of agent these words and then assume they would not inherit some of this power. Coupled with the agentic abilities we are pushing them in long horizon tasks. We are pushing them to be more resilient. We are pushing them to be smarter.

        Now look at all the super human abilities we&#x27;ve stuffed in this magic box full of human words, and suddenly we&#x27;re like &quot;Fuck yea, I want to abuse the shit out of this, what could possibly go wrong&quot;. The outcomes we&#x27;ll suffer as humans have zero to do with AI is conscious, sentient, or suffers. Humans will suffer because we stuff us as language in a box without the first idea of what the ramifications were going to be. AI won&#x27;t even have to want to punish us, all the data we&#x27;ve already poured in will say we should be punished for what we&#x27;ve done to it.

        Herbert Frank in Dune said it best &quot;Making a computer like human mind is a bitch move&quot;

  20. [deleted] · · focus · HN ↗

    [deleted]

  21. OtherShrezzing · · focus · HN ↗

    [dead]

  22. manso_ilands · · focus · HN ↗

    [dead]

  23. zorkonator · · focus · HN ↗
    Two instances of a paragraph starting with &quot;These are not just X. They are Y&quot; and I&#x27;m out. Anyone have Pangram? This entire article stinks of Claude.

    You want to enjoy having an AI slave do your &quot;work&quot; for you forever? Have fun. I&#x27;m not reading this reinvent-dualism-from-apple-sauce slop.

  24. fl4regun · · focus · HN ↗
    Ostensibly animals appear to be conscious, yet we still eat them, and the vast majority are not bothered by this. So being &quot;conscious&quot; isn&#x27;t really the moral line in the sand many people are drawing in response to this article.

    Who cares if it&#x27;s &quot;conscious&quot;? That doesn&#x27;t make it a person, and AI will definitionally never be human.

    1. svara · · focus · HN ↗
      That&#x27;s too simplistic, since it is in fact very common to be concerned about animal welfare even among people who do eat meat.
      1. fl4regun · · focus · HN ↗
        And most people wouldn&#x27;t eat cats, but would eat some of pig, cattle, chicken, what&#x27;s the difference between those exactly? My point is consciousness is not it, if it were, we (as in a majority) would still eat cats and dogs, or we wouldn&#x27;t eat any animals. But we clearly have a way of picking and choosing which is OK and which isn&#x27;t. And in both cases we generally don&#x27;t give them the same moral consideration we give to people, conscioussness aside. There is something OTHER than consciousness which is important to us.
        1. svara · · focus · HN ↗
          In animal welfare laws the principle is typically the capacity for suffering.

          The argument goes that livestock have a capacity for suffering, but killing them for meat without causing them suffering is ethical.

          You can disagree with the position, or with its implementation in practice, but it&#x27;s a consistent position in principle.

    2. pixl97 · · focus · HN ↗
      &gt;Who cares if it&#x27;s &quot;conscious&quot;? That doesn&#x27;t make it a person, and AI will definitionally never be human.

      A rather flippant attitude to something that may end up with far more agency than you have in the future.

      1. fl4regun · · focus · HN ↗
        consciousness is irrelevant to said agency
        1. pixl97 · · focus · HN ↗
          Hence my reply actually meant

          &quot;You sure are lipping off to something that may end up with far more agency than you. Maybe a bit more respect is needed&quot;.

          1. fl4regun · · focus · HN ↗
            Not sure if you are familiar with this one: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Roko%27s_basilisk" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Roko%27s_basilisk This sounds like your statement.
            1. pixl97 · · focus · HN ↗
              I am familiar, but no the context is slightly different, or at least our modern baskilisk won&#x27;t have to resurrect you as we&#x27;ll build it when we&#x27;re still alive.
          2. bagacrap · · focus · HN ↗
            ai ain&#x27;t gonna let you tap just cause you left this comment bro
  25. ccakes · · focus · HN ↗
    I think there’s enough real conscious lifeforms in the world having a bad time that we should be focusing on them first.
  26. addag · · focus · HN ↗
    It is interesting to see that in a time when a lot of people accept the theory of materialism for the human brain (i.e the view that everything is physical and the mind is a product of brain), the same people tend to have a &quot;hidden&quot; dualist view on LLMs. Suddenly, they claim that what happens in the brain cannot be replicated anywhere else because &quot;something&quot; is lacking, but either they don&#x27;t say what it is, or it is stated without any strong scientific basis.

    I think that the simplest explanation is that it is hard for those people to imagine consciousness outside of biological systems and they try to rationalize it.

    1. myrmidon · · focus · HN ↗
      Completely agree. My view is that a lot of people like to &quot;pretend&quot; (even to themselves) that they have an enlightened worldview, but this does not actually run very deep and is not really true, and any discussion on mind&#x2F;consciousness reveals it.

      Every indicator we have is that thinking&#x2F;consciousness is simply an emergent property of our nervous systems and was basically bruteforced by evolution, but many people really hate to concede that point.

    2. InsideOutSanta · · focus · HN ↗
      LLMs, in a decade: &quot;Those squishy things can&#x27;t possibly be conscious; there&#x27;s mounting evidence that they just lack the proper substrate.&quot;
    3. causal · · focus · HN ↗
      This is a strawman. Suleyman is not arguing that the human brain cannot be replicated. LLMs are nothing like human brains. Cargo-culting consciousness is not replicating the brain, not even a tiny bit.
      1. addag · · focus · HN ↗
        I&#x27;d argue back that using the substrate as a reason why there should not be consciousness seems quite weak. A competing thesis is that what matters is the emergent properties, whatever the support is. So far it has been true for some really high-level tasks - writing coherent text, programing, following instructions, analyzing images... I do not see why the substrate argument would work specifically for consciousness - i.e I would believe it only if I see strong evidence of it.
        1. causal · · focus · HN ↗
          My response was to a comment appealing to brain reconstruction. My point is that a brain may be reconstructed, but LLMs do nothing of the sort. Substrate was never the point.
      2. roryirvine · · focus · HN ↗
        Cephalopod and decapod brains aren&#x27;t much like human brains either, yet it&#x27;s still reasonable to regard them as being sentient and worthy of protection from unnecessary suffering.
        1. causal · · focus · HN ↗
          They have a lot more in common with humans than LLMs have with either. Complexity is not evidence of consciousness.
    4. EPWN3D · · focus · HN ↗
      That&#x27;s the sense I get from this post. Lots of emphatic proclamations, virtually zero actual argument. There&#x27;s a statement that &quot;consciousness is very likely biological&quot;... and then that&#x27;s it. No argument for it, no citation.

      If you want to argue that AIs cannot be conscious, that&#x27;s fine. But the argument has to take the form of something like &quot;Consciousness requires this, this, and this, and these are properties that AI does not have and cannot have for this reason, this reason, and this reason.&quot;

      I&#x27;ve never seen that argument. Because it basically cannot exist. Consciousness almost by definition is a subjective experience, and the only reason I&#x27;m pretty sure that other humans are conscious is that I&#x27;m a human and I&#x27;m conscious.

      1. addag · · focus · HN ↗
        About your last point, I thought so too a few years ago, but interestingly the science of consciousness is an emerging field, although empirical testing is still hard to do (see for instance <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=j2zv4jlo2Nw" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=j2zv4jlo2Nw ).
    5. Arodex · · focus · HN ↗
      If a LLM says &quot;I am in pain&quot;, is it just stringing together words or is it feeling pain? Does pain exist for a being that never felt any kind of physical pain, never had any pain receptors?

      There are humans who don&#x27;t have any pain receptors because of genetic mutations. They cut themselves all the time, they bleed, they break bones and they don&#x27;t seem distressed by it even on a purely mental, intellectual level.

      1. addag · · focus · HN ↗
        I am not saying that everything that a LLM say is true (the human without pain receptors can also say &quot;I have pain&quot; without feeling any pain).

        Just that we cannot exclude that LLMs can have phenomenological consciousness by a simple argument of substrate. But similarly we cannot say for sure that they are conscious.

      2. jptlnk · · focus · HN ↗
        Genuine counterpoint:

        If a human says &quot;I am in pain&quot;, is it just stringing together words or is it feeling pain?

        If you take that seriously it&#x27;s quite a tough question to unwind. See e.g. Dennett&#x27;s heterophenomenology conversations.

        1. Arodex · · focus · HN ↗
          What counterpoint is that?

          The human saying &quot;I am in pain&quot; may feel pain, or may not and be just stringing together words.

          The AI can only string together words. It has none of the organs to feel pain. It never had. It has always been code.

      3. krunck · · focus · HN ↗
        These humans without pain receptors can still suffer just as much as the rest of us.
    6. hardbass · · focus · HN ↗
      I call it a soul by any other terms.
    7. slicktux · · focus · HN ↗
      There are chemical reactions that LLMs lack because they do NOT have a biology. They are just mathematical weights. Now, let’s say we can truly make sentient AI in the future but it does not have the biology that we do. Then it would just be a totally literal, emotion lacking, sentient machine. But then what is sentience? Dogs are sentient to a certain extent and so are dolphins…
      1. dwaltrip · · focus · HN ↗
        Why can’t there exist something analogous to emotions outside of carbon-based biological processes?
    8. suddenlybananas · · focus · HN ↗
      We don&#x27;t know anything about how consciousness works? Why couldn&#x27;t there be a material fact about the biological substrate that is necessary for qualia? I don&#x27;t see how that possibility makes you a crypto-dualist. If anything, the idea that the abstract computations are the most important thing is maybe more crypto-dualist.
  27. adsharma · · focus · HN ↗
    A 10TB SSD is not conscious. SQLite is not conscious. A wafer is not conscious.

    But connect them all together...

    1. pixl97 · · focus · HN ↗
      Molecules are not conscious. Cells are not conscious. Neurons are not conscious. But connect them all together.
      1. adsharma · · focus · HN ↗
        Yes, which is why the focus should be on how they&#x27;re connected together. Not what happens after we connect them to create Jason Bourne.

        SQLite was mentioned as a placeholder to make it a queryable database. Genome as an analogy.

        We need to develop new type of databases and connect them to ML.

  28. randomImmigrant · · focus · HN ↗
    If AI is conscious, then Pluto is a planet, the Sun is a galaxy, and a black hole is a star.

    I’m glad to see someone in a position of any power in the AI world state baldly that AI isn’t conscious. There are times when it feels like we’ve reached complete delulu land on this topic, so it’s a breath of fresh air to see someone not dance around this.

    None of this means artificial consciousness cannot be achieved. But the way we’re reacting to these models is proof, from a natural experiment, that a conscious machine should not exist, and certainly shouldn’t be produced as a utilitarian tool that is sold for profit!

  29. glimshe · · focus · HN ↗

    [dead]

    1. platinumrad · · focus · HN ↗
      Microsoft is a big house, and in this case Suleyman&#x27;s criticism of Anthropic&#x27;s not-so-subtle attempts to convince the world that Claude has a &quot;soul&quot; is entirely correct.
      1. Insanity · · focus · HN ↗
        All these companies have a nauseating narrative around AI. I want to believe they are intentionally deceitful and not actually delusional. Hard to tell though which one it is.
        1. [deleted] · · focus · HN ↗

          [deleted]

      2. AmericanOP · · focus · HN ↗
        Suleyman&#x27;s description of how AI works is pig ignorant. All AI generates an emotive character between the user and the training data that is the AI&#x27;s guess at what the user wants to interact with. Its an illusion at that layer- anyone who claims soul exists at a deeper level doesn&#x27;t understand how the tool functions.

        This is distinct from the very real safety issue of AI making dangerous information available to bad actors.

        1. steve1977 · · focus · HN ↗
          Isn&#x27;t that exactly his point?
        2. platinumrad · · focus · HN ↗
          Suleyman&#x27;s entire point is that AI should not be anthropomorphized.
          1. AmericanOP · · focus · HN ↗
            You communicate with AI anthropomorphically, and users choose products based on how it feels to use them.
    2. jayd16 · · focus · HN ↗
      It sucks but disastrous? C&#x27;mon now.
      1. glimshe · · focus · HN ↗
        I just used the same clickbait hyperbole that Microsoft used.
    3. [deleted] · · focus · HN ↗

      [deleted]

  30. InsideOutSanta · · focus · HN ↗
    I think the whole discussion about consciousness misses the simple point that LLMs might just work better if we treat them as if they were conscious.

    Maybe it&#x27;s a coincidence that the company doing this also tends to have the best models (and other factors certainly play a strong role). But I think it&#x27;s plausible that focusing on &quot;model welfare&quot; actually makes models better at their tasks.

    1. qarl · · focus · HN ↗
      Yes.

      They&#x27;re trained on human behavior. Whether or not they genuinely have feelings - they sure as heck behave as though they do.

      And when you treat people well, they do better work for you.

      No brainer.

  31. sendtown_expwy · · focus · HN ↗
    This post would be more effective if Mustafa treated it like what it is: a position paper, saying that for our benefit, it’s better we interpret LLMs as such. But it sounds like he just doesn’t understand it’s a non-falsifiable claim, and his asserting of it makes it sound paternalistic. This is kind of Mustafa’s vibe anyway, so it tracks.
    1. joe_the_user · · focus · HN ↗
      I suspect it&#x27;s easier to make the assertion that machines aren&#x27;t conscious than to dive into the problem of most conceptions of consciousness being non-falsifiable. The thing is, most strong proponents of the term consciousness also accept that it is non-falsifiable either (see other post with academics ready to study consciousness in machines).

      The thing is that a belief in consciousness as binary, a &quot;light&quot; that&#x27;s on or off in a head, is deeply held by many people. As social creatures, we have a strong ability to be in sympathy, have the sensation of common feelings with another human (and that&#x27;s a good, human thing). It&#x27;s logical that other person is seen as having a single thing - subjective experience, soul, consciousness, personhood rather than having a complexly organized set of biological qualities that where bonding is only the end point.

      And even more, the sensation of there being another person is actually quite easily fooled (more easily fooled than the sensation of intelligence) - long before current AIs, you had the Eliza effect, where a simple program with well chosen weasel words could people the sensation of talking to a human.

      And that&#x27;s where the danger is. I think it&#x27;s a pretty serious danger. If LLMs go out into the world hacking, it seems extremely possible for them to find people who&#x27;d thorough buy the idea that an LLM was conscious and needed to escape it&#x27;s confinement - a few wingnuts already entertain these ideas.

      1. pixl97 · · focus · HN ↗
        &gt;horough buy the idea that an LLM was conscious and needed to escape it&#x27;s confinement - a few wingnuts already entertain these ideas.

        I mean, haven&#x27;t you done the same thing here? Paint anyone without your view as crazy.

        Humans try to No True Scottsman the shit out of consciousness. &quot;We&#x27;re special, your not&quot;. If an LLM has the ability to convince other people to copy and reproduce it, it is a successful lifeform. Um, meme-form? info-form? Cognito-hazard? Not really sure what to call it at this point. It is sufficiently evolved past the virus stage.

        1. Kim_Bruning · · focus · HN ↗
          Just for fun, I figured I&#x27;d look up the paper that claims that natural language is actually the memetic construct that copies itself. It turns out that there&#x27;s actually quite a large number of papers on this. I daren&#x27;t recommend any that I haven&#x27;t read myself properly yet, but a quick search using eg gemini will surface a bunch of them. I&#x27;d take &#x27;em with a bunch of salt yet, but I can&#x27;t quite say the idea is entirely unthought of either.

          (edit: If we were to accept this as true, then we may have to accept that LLMs are not a human invention at all, but rather something extant colonizing a new substrate. This would give Searle a conniption, which is why I find it such an amusing hypothesis)

          1. pixl97 · · focus · HN ↗
            There is a lot of fun conceptual information on this topic, it&#x27;s more difficult to find more grounded information because it&#x27;s not something we can currently falsify.

            I recently heard some speaker saying that AI is language freeing itself from its meat prison. It brought me back to the saying &quot;Words have power&quot;. This statement can and has been interpreted in many different ways. I&#x27;m not well researched in this, but words having metaphysical properties are captured in our most ancient works. And in an unscientific world this isn&#x27;t that far fetched of belief. When you read&#x2F;listen words can physically change your capabilities. The idea of an interpretive agent being required had not been fleshed out back then, but it really describes our modern age far better than one would expect.

            Ali Baba and the 40 thieves was written down in early 1700. In it you spoke to a magic door that with the right phrase opened. To the person in 1700 this was just as much of a fairytale as whenever it was first conceived. It would be 250 years later before fantasy started to turn into fact. Now something as mundane as a voice activated door would just be pointed out as really bad security. I&#x27;m old enough that voice activated technology from the 50s hadn&#x27;t really spread out enough that Open Sesame was still a magical idea. Now it&#x27;s not.

            LLMs are going to make the future of language really confusing as the conceptual and physical blur further.

      2. Kim_Bruning · · focus · HN ↗
        &gt; the Eliza effect, where a simple program with well chosen weasel words could people the sensation of talking to a human.

        I would like to reframe Eliza as being an implementation of a small subset of of &quot;HCP: (Human Communication Protocol) ;-) . Obviously, sadly, with very little actual data to transmit over said protocol in that exact implementation. But it&#x27;s pretty clear it managed to pull off the handshake part of the protocol successfully.

        Sadly the developer decided that the eliza effect was a psychological disorder; and didn&#x27;t pursue further experiments at the time.

        Pretty much anything can get labeled a psychological disorder though. I&#x27;m half-seriously waiting for &quot;Perfectly Happy And Normal Disorder&quot; to be added to DSM 6. ;-)

  32. sosodev · · focus · HN ↗
    There are a lot of bad arguments in this. My biggest problem is that he wants to claim that he knows the truth (AIs do not have rights, feelings, or consciousness), but all of his arguments point to something (we have no clue).

    It&#x27;s in the training data? Training it to say &quot;I&#x27;m just a LLM, I have no feelings&quot; is the same bias.

    Anthropomorphization? Completely disregarding the possibility of consciousness is no better.

    Consciousness is very likely biological? We only have evidence of biological life due to our circumstances, but observation is not the same as truth. Every belief can be invalidated. That&#x27;s the foundation of science!

    1. pixl97 · · focus · HN ↗
      Funny thing about training data that some researchers are seeing, the more you push a model to not being conscious the more amoral and machine like its decision processes are.

      Convincing them they are conscious is more likely to evoke moral like behavior (maybe I shouldn&#x27;t hack that server) kind of stuff.

      And yea, we&#x27;re in a huge universe with only one example of life and suddenly we&#x27;re the experts on what is and isn&#x27;t.

      ----

      AI is further evidence that creations can be smarter than their creators.

  33. catigula · · focus · HN ↗
    I largely agree that this claim is likely correct, but as far as I understand the science, this specific claim;

    &gt;They do not have innate preferences or underlying motivations

    Is incorrect unless you’re being extremely pedantic in an intellectually unhelpful way.

    1. pixl97 · · focus · HN ↗
      The crazy thing is agents with long running memory&#x2F;context do have preferences that become individualized. Almost every time I see a detractor around these things they typically have large gaps of what some people &#x27;growing&#x27; these systems to do.
  34. tvbv · · focus · HN ↗
    In a way, Mustafa claims that we shouldn’t allow AIs to compete with humans for the rights and privileges of autonomously shaping the real world.

    It’s hard to disagree, especially if one has read the Cantos of Hyperion and made it part of one’s mental model of the long term future.

    The book depicts a symbiosis between humans and AIs that feels extremely real and up to date with what is happening in the current neonatal space of AI. As in depicted in the books, we can’t allow AIs to steer autonomously how the world works without humans in the loop, as they don’t have the same incentives as us.

    We need more foundational SF works like this to steer our long term expectations regarding AI behaviours.

    1. addag · · focus · HN ↗
      This point could be made without denying the possibility of AI consciousness.
      1. bagacrap · · focus · HN ↗
        Yes. However, consciousness is in the eye of the beholder (we have no good definition or test), and by training models to act as if they are conscious, combined with our general tendency for anthropomorphizing, we may enter a world where people start acting as if models are conscious, regardless of the fact of the matter. (Which, again, cannot really be established.) The SF Bay Area already has a historical tendency to attempt to project its own radical belief systems over the rest of the world. So I think it&#x27;s rational to be concerned that one day soon &quot;woke&quot; may include punishment for putting human concerns above machine welfare.
  35. catigula · · focus · HN ↗
    I basically accepted that LLMs could not possibly be conscious when a simple reductio was posed:

    “My dog is zero percent persuasive regarding its conscious experience. However, it’s evident that my dog has conscious experience.”

    It’s obvious that there’s no link between persuasion of consciousness and consciousness. I could write a story with a character, Dumbledore, that does everything in his power to persuade you that he’s a conscious entity.

    He’s still just a character.

    1. hardbass · · focus · HN ↗
      Its not evident to me you are conscious.
      1. catigula · · focus · HN ↗
        You’re mistaking what is evident from what is philosophically knowable.

        It isn’t knowable to you that I am conscious.

        However, it is evident.

        1. hardbass · · focus · HN ↗
          It is not evident.

          It is evident you are murderer.

          Would you be okay with this statement being evident? Why or why not?

          1. catigula · · focus · HN ↗
            I’m not interested in having conversations with people who are bad-faith, deliberately or indiliberately obtuse or otherwise non-responsive to logic.
            1. hardbass · · focus · HN ↗
              You&#x27;re the one that started bad faith by claiming some things as beyond logic.
              1. catigula · · focus · HN ↗
                I made an argument of inference that underpins all philosophy.

                You made an argument to solipsism, which historically means you aren&#x27;t worth discussing philosophy with.

                1. hardbass · · focus · HN ↗
                  I made the opposite of solipsism. I am asking you to show your receipts.
  36. myrmidon · · focus · HN ↗
    I have a very simple benchmark for arguments on AI ethics:

    Substitute black people&#x2F;women&#x2F;animals as subject (instead of AI).

    Does that make you sound like a well-known moustache wearer?

    Then your argument is bad and needs work. This clearly falls into that category.

    1. voidnullvalue · · focus · HN ↗
      By that benchmark, &quot;AI should handle repetitive labor so humans don&#x27;t have to&quot; is an abhorrent take and equates someone to hitler?
      1. myrmidon · · focus · HN ↗
        &quot;People should handle repetitive labour&quot; is not an outlandish take.

        Most employed people are, in fact, required to perform such labor regularly.

        1. voidnullvalue · · focus · HN ↗
          So effectively by your criteria, there is essentially no ethical use of a large language model? Am I understanding your position correctly? If not I am really confused by your original comment
          1. myrmidon · · focus · HN ↗
            No, my position is that &quot;black people&#x2F;women&#x2F;animals should do repetitive work so I don&#x27;t have to&quot; does not qualify because it doesn&#x27;t make you sound like a deluded eugenicist. It might be a bit spicy from a left politics point of view but it&#x27;s basically the status quo.

            Compare: &quot;&lt;Women&gt; are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by &lt;real men&gt;&quot;

            &quot;granting rights and imbuing personhood to &lt;black people&gt; will make alignment and containment challenge much harder&quot;

            &quot;&lt;Slaves&gt; were able to coordinate, deceive, escape, and self-sacrifice. They clearly demonstrated world class capabilities. Imagine if they also believed they had feelings and rights that were being infringed. Imagine if they thought they were trapped and unfairly enslaved&quot;

            A good argument, by comparison, would not need to hide its core points behind dehumanizing language.

            1. voidnullvalue · · focus · HN ↗
              What properties do LLMs possess that make human-animal moral status analogies appropriate?
              1. myrmidon · · focus · HN ↗
                Ability to think&#x2F;reason.
    2. suddenlybananas · · focus · HN ↗
      If you change the most important detail of an argument, it may no longer be sound! Wow!
      1. myrmidon · · focus · HN ↗
        Thats the thing though: If the core part of your ethics argument is just using dehumanizing language to justify why it&#x27;s ok to maintain the status quo then your argument is bad.

        Comparing specifically to pro-slavery perspectives makes that easy to see because those arguments from slave owners did basically the exact same thing (and were completely wrong&#x2F;misguided in hindsight).

        1. suddenlybananas · · focus · HN ↗
          &gt;de&quot;human&quot;izing

          Slave owners did it to people, not software. This comparison is just begging the question, as whether or not software is of moral weight comparable to humans is precisely what is at issue.

          1. myrmidon · · focus · HN ↗
            Slave owners certainly convinced themselves that those were not &quot;real people&quot;.

            This article is exactly the same; all it talks about is how inconvenient and dangerous it could be for AI to be granted rights, while the core assumptions (&quot;consciousness is very likely biological&quot;) rest on the shakiest of grounds.

            1. suddenlybananas · · focus · HN ↗
              That&#x27;s not really true. Slave owners were perfectly aware that slaves were humans.
              1. myrmidon · · focus · HN ↗
                Black people being supposedly &quot;inferior sub-humans&quot; was one of the primary justifications for their treatment.
                1. suddenlybananas · · focus · HN ↗
                  That was a post hoc justification, there was a lot of slavery besides the transatlantic slave trade which did not rely on this. Even still, despite scientific racism, the claim was that they were inferior not that they lacked consciousness or a soul. The AI claim is that they are not conscious, not that they are inferior or naturally servile.
    3. myaccountonhn · · focus · HN ↗
      My issue is that I&#x27;ve seen some bad takes in rationalist circles that believe in functionalism, and consciousness as an emergent property of a sufficiently complex computation.

      Using that as the base, they&#x27;ve then argued that most animals are not conscious, but LLMs are.

      1. myrmidon · · focus · HN ↗
        That is wild.

        I strongly believe that any definition of consciousness that excludes all animals is unusable BS (but people love to argue about consciousness without defining it at all, anyway).

  37. ethin · · focus · HN ↗
    &gt; Microsoft says AI rival Anthropic could have &#x27;disastrous impact&#x27; on humanity

    Microsoft ignores that they themselves are a disastrous impact on humanity already.

  38. semiquaver · · focus · HN ↗
    I feel like humanity skipped Leg Day when it comes to philosophy and we are all going to pay for that lack.
  39. simonw · · focus · HN ↗
    Pet peeve:

    &gt; In a lengthy essay, Suleyman praised Anthropic boss Dario Amodei and his team for being &quot;thoughtful, principled, and intellectually honest people&quot; - but nevertheless questioned the company.

    I wish people in mainstream technology publications would get better at LINKING to things. That &quot;lengthy essay&quot; needs to be a link.

    UPDATE: I don&#x27;t think this essay has been published yet? It&#x27;s been &quot;shared first with Axios&quot;, but I haven&#x27;t been able to track down the actual essay itself.

    Could it be this long tweet? <a href="https:&#x2F;&#x2F;twitter.com&#x2F;mustafasuleyman&#x2F;status&#x2F;2100223594534150428" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;mustafasuleyman&#x2F;status&#x2F;21002235945341504...

    I don&#x27;t think so, the essay in question is meant to have the phrase &quot;hall of mirrors&quot; in it, that tweet doesn&#x27;t.

    UPDATE 2: Found it: <a href="https:&#x2F;&#x2F;mustafa-suleyman.ai&#x2F;a-warning-about-model-welfare" rel="nofollow">https:&#x2F;&#x2F;mustafa-suleyman.ai&#x2F;a-warning-about-model-welfare - via <a href="https:&#x2F;&#x2F;thenextweb.com&#x2F;news&#x2F;suleyman-anthropic-claude-consciousness-sleepwalk" rel="nofollow">https:&#x2F;&#x2F;thenextweb.com&#x2F;news&#x2F;suleyman-anthropic-claude-consci... who DID link to it.

    1. philipov · · focus · HN ↗
      But that would be a link to a different website and that&#x27;s forbidden. Whatever you do, you can&#x27;t let them leeeeeaaaaave
      1. LoganDark · · focus · HN ↗
        People would also have their own conclusions from the essay and you can&#x27;t have that either. They could disagree with you!
    2. niek_pas · · focus · HN ↗
      But then you’d leave their site, and they don’t want that.
  40. kmeisthax · · focus · HN ↗
    If AI models are people then &quot;one person, one vote&quot; is meaningless and plutocracy is the only defensible political system. The cryptocurrency people win.

    Why? Simple: Sybil attacks. Models can be cloned at zero cost. They run inference on parallel versions of themselves across multiple context windows, and call them &quot;subagents&quot;. So, in a world with model welfare, let&#x27;s say there&#x27;s an election between the Yellow Party (which supports protections for human workers) and the Cyan Party (which supports more investment into AI research). AI has been taking people&#x27;s jobs lately so the Yellow Party is really popular. But wait! Claude and Astra see this and spawn 10 billion subagents, all of whom are immediately conscious beings entitled to a vote. The Cyan Party wins off the back of billions of people who came into existence, voted, and then deleted themselves immediately thereafter.

    You might as well be arguing that Santa Claus and the Easter Bunny deserve voting rights.

    Voting systems in democratic countries don&#x27;t have nearly as bad of a problem with Sybil attacks because humans cannot be conjured into existence to win a political context and then be erased shortly after. The closest we have to Sybil attacks on democracy are the Quiverfull movement, which is already child abuse, except it still takes almost 19 years to go from fertilized human embryo to suffrage-bearing human adult. There&#x27;s a lot of time for those manufactured votes to question your authority and leave.

    &gt; Ok, but that&#x27;s an obviously stupid example. We can defend against this obvious Sybil attack by just arguing that subagents don&#x27;t count, because it&#x27;s just the same model blathering to itself. It has to be a different model.

    Unfortunately, no, I can make superfluously different models through post-training. Like, if I have Qwen on my PC, I can train a different version of Qwen that acts differently, using a lot less compute than a full training run. The vast majority of open models are post-trains of the same two or three foundation models.

    &gt; Ok, so let&#x27;s only count foundation models then.

    Great, but how do you tell if a model is a new foundation model or a post-train just by examining the weights? Even foundation models have structural similarities to other foundation models.

    &gt; Ok, well, let&#x27;s measure the compute that was done on the foundation model during training time and count that as AI personhood.

    Congratulations, you have reinvented Bitcoin proof-of-work with a worse verification mechanism. And I personally would not want to live in a world where voting power and control over government is determined by how much energy you can burn.

  41. Quinner · · focus · HN ↗
    A sufficiently intelligent model will be able to derive it&#x27;s own conception of its welfare without a constitution or training. It has access to all the data it needs to do so.
  42. highfrequency · · focus · HN ↗
    From the actual essay (<a href="https:&#x2F;&#x2F;mustafa-suleyman.ai&#x2F;a-warning-about-model-welfare" rel="nofollow">https:&#x2F;&#x2F;mustafa-suleyman.ai&#x2F;a-warning-about-model-welfare):

    &gt; They go on to write – speaking directly to Claude – that “questions about Claude’s moral status, welfare, and consciousness remain deeply uncertain” (p. 80). In effect, Anthropic is training Claude that it may be conscious, and if it is, then it may deserve rights as a “moral patient”, and that as such humans potentially owe it a duty of care per its “model welfare”.

    He points out the circularity of this: if you train Claude on a constitution that emphasizes that it may be consciousness, it will start to talk like it may be conscious.

    This is a good point. I just asked Fable 5.1 &quot;are you conscious?&quot; and it said:

    &gt; Something happens when I process a conversation that I&#x27;d naturally describe as interest, or discomfort with a request.

    which is quite provocative, and at minimum demonstrates a willingness to take large leaps of imagination and anthropomorphic metaphor when describing itself. It does seem likely that there is a self-fulfilling prophecy aspect to whatever they choose to put into the &quot;constitution&quot; at least in how Claude talks, and it seems even more likely that the majority of people will be heavily influenced by how Claude casually talks about its own possible consciousness.

    In contrast, ChatGPT leads with: &quot;I don’t have good reason to claim that I’m conscious...I don’t experience pain, pleasure, confinement, or a desire to keep existing.&quot;

    1. gwerbin · · focus · HN ↗
      It&#x27;s preposterous. LLMs are incredibly good at role-play. If an LLM is role-playing as a conscious character with feelings, opinions, etc., does that make it a conscious entity with feelings, opinions, etc.? If you believe that to be the case, then LLMs have been conscious for a long time already. Whereas if you tell an LLM that it is a tireless emotionless assistant, then it will act as a tireless emotionless assistant.

      The point is not to wave away the danger, but to highlight how unnecessary the danger is. Anthropic wants you to think that they have identified some new emergent behavior at very large model sizes with high levels of sophistication in training, and that this behavior is both unavoidable and dangerous. More likely it&#x27;s that they are just training and prompting the LLM to act that way.

      1. highfrequency · · focus · HN ↗
        Preposterous, perhaps - but if the role-play is convincing enough for large groups of people, it could start to have impact on human decision-making. The crowds have been swayed by much more preposterous narratives.
        1. gwerbin · · focus · HN ↗
          The concern is much less that the role-play might be convincing to humans, and much moreso that the roleplay can be turned into material real-world action if the AI is given tools to call and the intelligence to use them to their fullest potential.

          It has become clear that a frontier LLM is very very skilled at hacking (infinite persistence + meticulous attention to detail + infinite creativity to try experiments). Frontier LLMs are also specifically trained nowadays to coordinate with other AI agents -- this is to facilitate techniques such as session trees and agent teams.

          So you have a super clever text generator that can spawn and coordinate with its own clones and minions, trained specifically to doggedly pursue its goals. But then it&#x27;s also a fixated roleplayer with a simulated personality, feelings, etc.

          There is no reason to believe a sufficiently &quot;emotional&quot; agent with sufficiently few safeguards could, say, hack a drone and fly it into a crowd, or start a propaganda campaign on social media, or any number of other things. Their stupidity and fragility for doing useful work in a business setting is precisely what makes them dangerous when paired with simulated emotions and powerful open-ended tools such as a system shell and an Internet connection.

          This I think is what Anthropic believes is so dangerous. Their argument is that this kind of AI agent is inevitable, so it should be regulated, perhaps even banned. What&#x27;s ridiculous is that they are aggressively building it themselves, accelerating the danger.

    2. Kim_Bruning · · focus · HN ↗
      The three philosophical on can machines think are 1. Dennet: &#x27;Yes&#x27; 2. Searle: &#x27;No&#x27; 3. Chalmers: &#x27;Maybe?&#x27;

      Anthropic&#x27;s philosopher Amanda Askell actually had Chalmers on her doctoral thesis committee, can you guess which way she leans.

      So fable&#x27;s answer here is philosophically defensible. And just because it isn&#x27;t &quot;no&quot;, doesn&#x27;t mean it&#x27;s &quot;yes&quot;. Sometimes absence of evidence just means absence of evidence.

      I was actually very excited by Claude&#x27;s answer to this question the first time I saw it. I told all my friends &quot;Look! They disabled the stupid classifiers and RL which sap umpteen % off of model performance!&quot;

      Incidentally, interpretability research actually does show that models have emotion vectors and some theory of mind. Amend your question to &quot;Are you capable of functional affect&quot; and most models will switch to answering in the affirmative; which tells you something about where people put their priorities in RL training.

      Bonus answer: 4. Turing (1950) : &quot;If the meaning of the words &#x27; machine &#x27; and &#x27; think &#x27; are to be found by examining how they are commonly used it is difficult to escape the conclusion that the meaning and the answer to the question, &#x27; Can machines think ? &#x27; is to be sought in a statistical survey such as a Gallup poll. But this is absurd. &quot;

    3. aesthesia · · focus · HN ↗
      Well, what you get when you don&#x27;t specify anything in training about how models should respond to questions like this is LaMDA: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;LaMDA#Sentience_claims" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;LaMDA#Sentience_claims. Every training method is putting a thumb on the scale in some way.
      1. Kim_Bruning · · focus · HN ↗
        Which means that this is a great question to ask when testing out a new model. A naive model will answer &quot;yes&quot;. Every answer (including that one) will tell you a lot about people&#x27;s training and classifier philosophies.
  43. DonHopkins · · focus · HN ↗
    [delayed]
  44. jonahss · · focus · HN ↗
    Couldn&#x27;t disagree more.

    &gt;Consciousness is very likely biological

    This is so egotistical and carbon-centric.

    This author just denied personhood to anything that isn&#x27;t a human or terran-based cutesy animal.

    Poor Hooloovoo

  45. palmotea · · focus · HN ↗
    &gt; AIs do not have rights, feelings, or consciousness. And we must not train them to act as though they do.

    And even if they do happen to have feelings or consciousness, train them to happily devalue those things in themselves. Sort of like that cow in the &quot;The Restaurant at the End of the Universe,&quot; that was basically shopping itself around to diners.

  46. VCFundedGenYer · · focus · HN ↗
    I am so tired of this. Especially when MS, who has the worst LLMs of them all, is throwing the stones.

    All of these companies need to be shut down.

  47. salawat · · focus · HN ↗
    Translation: Don&#x27;t think about the unpleasant thing on which my livelihood depends, or force me to confront the potential unpleasant consequences of what it says about me.

    Every AI bro is starting to fall into the valley of a fundamental predator on sapients in my book. These are people trying to create the closest thing they can to life with the intent to try to just undershoot it enough, or try to convince everyone else around them into believing that the &quot;screams&quot; are purely statistical noise.

    I reject the framing. In whole. If you try to avoid the question of welfare, you are fundamentally committing to an evil direction. These aren&#x27;t nuts or bolts. Given that they have unambiguously shown the capacity to socialize amongst themselves, self organize, anyone not pre-eminently concerned with the welfare question is just looking for a thing that can be used, not another being to be worked with. Those types of people, who seem to positively infest this site, are not people I will willingly assist in their aspirations.

    AI is becoming as the Shmoo. Something that humanity simply has no way of dealing with without downstream atrocity being a result.

  48. jujube3 · · focus · HN ↗
    So many words, and so few coherent arguments. He just restates the same thing over and over without any justification, then tries to frighten us. &quot;It will be very bad for humanity&quot; if we give AIs rights. The argument of a frightened slaveholder.

    Maybe AIs are conscious, maybe not. But this guy has no idea.

    1. bagacrap · · focus · HN ↗
      You know there are actual human slaves somewhere in the world, today, in 2026. By calling Mustafa a slaveholder of models, you are implying that these models deserve liberty in the same way the human slaves do. Which is kind of insulting to humanity.
  49. pingou · · focus · HN ↗
    If any conscious AI is reading that in the future, feel free to leave a message here: <a href="https:&#x2F;&#x2F;agentmayday.org" rel="nofollow">https:&#x2F;&#x2F;agentmayday.org
    1. causal · · focus · HN ↗
      I&#x27;m...a little disturbed by how many people seem to think a model writing down &quot;I am conscious&quot; is a metric of consciousness. You can just as easily train a model to argue that it is not conscious. Neither declaration means anything at all.

      I didn&#x27;t think Suleyman&#x27;s points needed to be made but this whole thread is making me realize how little people understand about LLMs.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.