‹ BackHN Continuity

Thread

A warning about 'model welfare'

242 points · 701 comments · andsoitis

  1. qarl · · focus · HN ↗
    Birch, The Edge of Sentience (2024), ch. 16 - "simply no way to assess sentience in an LLM"

    Schwitzgebel, AI and Consciousness (2025) - "we won't know before we've already manufactured thousands or millions of disputably conscious AI".

    Butlin, Long et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) - "no obvious technical barriers to building AI systems which satisfy these indicators".

    Chalmers, Could a Large Language Model Be Conscious? (2023) - "within the next decade, we may well have systems that are serious candidates for consciousness".

    Long, Sebo, Butlin, Birch et al., Taking AI Welfare Seriously (2024) - "there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future".

    Dreksler, Caviola, Chalmers, Sebo et al., Subjective Experience in AI Systems: What Do AI Researchers and the Public Believe? (2025) - survey of 582 AI researchers; median estimate of 25% by 2034, and only 10% that such systems will never exist.

    1. goodmythical · · focus · HN ↗
      We cannot test for that which we cannot define. Given that we cannot rigorously define sentience, we cannot test for it. Doesn't really matter whether we're talking about people who are locked in comas, "brain-dead" individuals, dolphins, primates, dogs, or the carefully polished and arranged minerals that we call processors.

      There are those who believe that were they reduced to life support, they would no longer be alive and should therefore not be supported by said machines.

      There are those who believe that penguins, dolphins, eagles, and more are sentient beings that make choices understanding the consequences, develop love of their partners and mourn their losses, and feel, display, and act upon their emotions.

      There are those who believe that fungi/trees/plants are either individually sentient or sentient as a part of a network. Choosing to sacrifice their own nutrients to answer the call of a wounded neighbor, for instance.

      Although, there are also those who believe that human's don't have any special unique quality that isn't shared by either all living things or all things in general. These individuals already believe that the machines have the same kinds of qualities as we do. They are slow when they are unhealthy (needing a dusting or coolant loop bleeding being equivalent to us needing some fresh air for instance) and uncooperative when upset (by a virus, full hard drive, or oom).

      1. throw310822 · · focus · HN ↗
        > We cannot test for that which we cannot define.

        That's not the point. You know exactly how to define being conscious and aware- it's your subjective experiencing of the world and of your inner states. The problem is that being a subjective experiencing, there is no way to communicate it to the outside world.

        1. Loquebantur · · focus · HN ↗
          That's obviously incorrect, as humans routinely talk about their subjective experiences.

          Maybe that's not as common with HN folks, but most other humans do.

          1. pixl97 · · focus · HN ↗
            No, humans output a stream of tokens that sound like what a being with subjective experiences would output. Both you and I make an assumption that this stream of words is at least somewhat representative of what the are experiencing and that the person is not a p-zombie.

            But you have zero proof that their words aren't a confabulation, you can only know than when you output a similar set of words you had a subjective mind state they represent.

          2. loopies · · focus · HN ↗
            Even so you still cannot ever test for it.

            The very best you can do is define human consciousness as requiring the functions of a human brain, and try to figure out (decide/agree) the chances that something which is working loosely on the same principles is or isn't conscious.

            At the end of the day it will always be an agreement, never scientifically proven as fact.

          3. mitxela · · focus · HN ↗
            throw310822 is actually a Markov chain that just happened to output those sentences. Markov chains are definitely not conscious.
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.