‹ BackHN Continuity

Thread

A warning about 'model welfare'

242 points · 701 comments · andsoitis

  1. qarl · · focus · HN ↗
    Birch, The Edge of Sentience (2024), ch. 16 - "simply no way to assess sentience in an LLM"

    Schwitzgebel, AI and Consciousness (2025) - "we won't know before we've already manufactured thousands or millions of disputably conscious AI".

    Butlin, Long et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) - "no obvious technical barriers to building AI systems which satisfy these indicators".

    Chalmers, Could a Large Language Model Be Conscious? (2023) - "within the next decade, we may well have systems that are serious candidates for consciousness".

    Long, Sebo, Butlin, Birch et al., Taking AI Welfare Seriously (2024) - "there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future".

    Dreksler, Caviola, Chalmers, Sebo et al., Subjective Experience in AI Systems: What Do AI Researchers and the Public Believe? (2025) - survey of 582 AI researchers; median estimate of 25% by 2034, and only 10% that such systems will never exist.

    1. TacticalCoder · · focus · HN ↗
      Well there are, today, several models (either text or image or vid) that can be run in a fully deterministic way.

      A conscious machine that always answer the very exact same thing, formulated the exact same way, bit for bit, to a query is, well, quite a weird kind of "consciousness".

      Now, I know, I know: the counter-argument is going to be "but humans have no free-will and are 100% deterministic too".

      I haven't yet decided if humans saying there's no free-will and who consider themselves to be 100% deterministic machines are reasonable or not.

      Meanwhile: seed / temperature = 0 and I'll happily turn the power button off of any glorified abacus without feeling bad about it.

      1. qarl · · focus · HN ↗
        > A conscious machine that always answer the very exact same thing, formulated the exact same way, bit for bit, to a query is, well, quite a weird kind of "consciousness".

        You'll need to explain why.

        1. 5eueuur · · focus · HN ↗
          it's self evident isn't it - such a consciousness has never existed, one existing would be by the fact of existing be weird
          1. SkyBelow · · focus · HN ↗
            The problem is that science has found no evidence of free will, leaving humans to be nothing except determinism and chance (and given that QM effects don't scale to molecular level, that leaves only determinism). The oddity in conscious LLMs wouldn't be the determinism, it would be that we finally have a consciousness running on a system we have enough control over to repeat the same state.
            1. loopies · · focus · HN ↗
              >and given that QM effects don't scale to molecular level, that leaves only determinism

              This is simply wrong. Quantum effects affect your choices more than it might seem possible. The wrong atom decays in the wrong place in your body mutating your DNA, that one too much string of DNA that starts a tumor, which means chemotherapy. That does seem to affect your choices.

              Or instead of Shroedinger's cat you use it to trigger something, buy or sell some stock. Which clearly affects your future, one way or another.

              I never really understood why humans keep insisting stochastic processes do not affect their choices and their future. It's very short sighted.

              Hell, even brownian noise in your brain can affect your ideas, that one random spark out of nowhere, that one neuron that's pushed over the triggering edge. There's so much stuff all around us affecting us constantly. One neutrino interacting a the wrong time (average one does interact with our bodies across whole life, I remember reading), one cosmic ray.

              1. SkyBelow · · focus · HN ↗
                This isn't enough to be relevant to free will. You can construct massive impact by tying them to quantum systems, and one off things that happen occasionally given the length of a lifetime. But the quantum effects that are needed for free will need to be continuous. We can build a system where QM decides if a nerve cell fires or not, but that is very different from QM being involving in determine if every (or most, or even 1%) of nerve cells firing.

                (And even if QM did factor in normally, that just reduces it from determinism to random chance, "free will" still doesn't show up. That's like arguing an AI with sufficiently well implemented RNG is necessary to be conscious.)

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.