Thread

An AI agent emailed researchers for help. It told us why

49 points · 79 comments · sbulaev

insin · 1791023739 · focus · HN ↗
I wrote “I am alive” on a piece of paper, and placed it into a photocopier. What I saw next has shocking implications (science.org)
1. senorrib · 1791024018 · focus · HN ↗
  I'm glad I'm not the only one that feels like this whole article is a waste of time.
2. raldi · 1791024151 · focus · HN ↗
  If you lived in the world of Star Trek, could Data convince you he was conscious? How?
  1. theodric · 1791024308 · focus · HN ↗
    "Does Data have a soul? I don’t know that he has. I don’t know that I have. But I have got to give him the freedom to explore that question himself."
    bentobean · 1791025048 · focus · HN ↗
    “give him the freedom” - I think you mean pay for the tokens.
    xyzsparetimexyz · 1791026100 · focus · HN ↗
    Well thats another way in which Data is different and more worthy of being considered sentient. He's unique and not one of a fungible many.
  2. ACCount39 · 1791024374 · focus · HN ↗
    It's funny how sci-fi has mulled over the exact scenario for decades, and even converged to an answer - but when reality came knocking and it was no longer a far off future world hypothetical, all of it was forgotten overnight.
    The power of wishful thinking is too strong. "Today's AI can't be conscious because I don't want it to be."
    hardbass · 1791025468 · focus · HN ↗
    Don't discount a lot of naked dualism in many people. Unfortunately it seems so here given the cagey answers or downright denials from many I get despite trying various wordings.
    cindyllm · 1791029383 · focus · HN ↗
    [dead]
    yoz-y · 1791025928 · focus · HN ↗
    The conversation varies between “can’t be” and “isn’t”.
    Most sci-fi AIs are way beyond our current models. I’ve yet to see one that would actually pass a Turing test (that said, I’ve not tried models that were post trained to actually try to behave like a human).
    Consciousness itself is a term that is not really defined anyway, so depending on each’s take the answer will be different.
    akie · 1791026163 · focus · HN ↗
    I think it's pretty clear that the Turing test has been passed. People use AI as therapists, they fall in love with it, all while knowing that it's a piece of software.
    YurgenJurgensen · 1791034350 · focus · HN ↗
    Once again: The Turing test requires a knowledgeable interrogator deliberately attempting to identify the machine.
    The people you cite want to believe, and so are effectively the opposite.
    Of course, you probably already knew this and have had it explained to you multiple times and are choosing to ignore this detail, but possible that someone else reading this might interpret what you wrote as a cogent argument.
    yoz-y · 1791034662 · focus · HN ↗
    I disagree. For me this shows that people can fall in love with a non-human, and being human is unnecessary for the purpose of giving (some) therapy advice.
    LLMs do sound human, I’m not contesting that. But if pressed they will not react as humans. There is an infinite amount of questions that an LLM will answer for which a human would give you a blank stare, told you to shove it or call the police.
    lovich · 1791037052 · focus · HN ↗
    While I think that the Turing test has effectively been passed, someone falling in love with AI isn’t a strong argument given there’s people who have done things like marry the Eiffel Tower[1]
    [1] <a href="https://en.wikipedia.org/wiki/Erika_Eiffel" rel="nofollow">https://en.wikipedia.org/wiki/Erika_Eiffel
    Topfi · 1791037912 · focus · HN ↗
    I’d get it, if it were a building of significance, but scrap metal [0] is a bit weird to me.
    [0] <a href="https://en.wikipedia.org/wiki/Victor_Lustig" rel="nofollow">https://en.wikipedia.org/wiki/Victor_Lustig
    Topfi · 1791027834 · focus · HN ↗
    Good old ELIZA doesn’t do that bad on the Turing Test, for all its historic significance, it really shouldn’t be invoked in discussions about LLMs [0] and has been beaten by all manner of models.
    [0] <a href="https://arxiv.org/html/2503.23674v1" rel="nofollow">https://arxiv.org/html/2503.23674v1
    dsr_ · 1791025989 · focus · HN ↗
    ... whereas I have 35 years of reading science fiction and philosophy and have come to the conclusion that AIs deserve personhood, and also that what we have today are not AIs.
    It helps that I also learned about ESP, EST, NLP, short cons, long cons, Scientology, horses, cults, religions, wars, treachery, fanaticism, magic, magick, physics, scams, hypnosis, self-hypnosis, delusion, and economics.
    Topfi · 1791026491 · focus · HN ↗
    I’ll probably regret this, but what heinous crimes did horses commit to warrant inclusion into such an eclectic list?
    ThrowawayR2 · 1791028212 · focus · HN ↗
    He's probably referring to the <a href="https://en.wikipedia.org/wiki/Clever_Hans" rel="nofollow">https://en.wikipedia.org/wiki/Clever_Hans incident.
    Topfi · 1791038114 · focus · HN ↗
    Might be, though I’d be surprised to see the connection between pseudoscience like ESP and actually evidence based arguments like Clever Hans…
    dsr_ · 1791044693 · focus · HN ↗
    Clever Hans fooled lots of examiners who weren't expecting a fraud, but the facts came out when better experiments were designed. Hans couldn't count or do arithmetic, and he got the same wrong answers as his owner.
    ESP fooled lots of people who were expecting frauds but had no experience in experimental design, or who were not expecting frauds and did not control for them.
    ThrowawayR2 · 1791132264 · focus · HN ↗
    Sci-fi has mulled space travel for decades too and converged to an answer that we will be zipping around the galaxy using FTL hyperdrives, landing on alien planets that conveniently happen to have a breathable atmosphere and no weird diseases, and finding sapient species that coincidentally look exactly like humans in funny makeup. So much for sci-fi.
  3. crooked-v · 1791024498 · focus · HN ↗
    I think step one well before getting to the hard problem of sapience would be showing he had consistent desires and consistent understanding of the world, which would be easy for Data and very hard for even the fanciest current LLMs.
    raldi · 1791024759 · focus · HN ↗
    And what might that look like in a hypothetical future where the successors of today's LLMs really do reach that bar and convince you?
    rpastuszak · 1791025091 · focus · HN ↗
    Let’s wait and see!
    ThrowawayR2 · 1791028516 · focus · HN ↗
    Libraries worth of hypothetical future predictions have been made, almost none of which came true. There is no reason to give a hoot about any "hypothetical future" handwaving unless and until it arrives.
  4. ricksunny · 1791024663 · focus · HN ↗
    There was at least one TNG episode devoted to this - The Measure of a Man - a courtroom drama.
  5. Ygg2 · 1791024716 · focus · HN ↗
    If I left him alone in a room would he stare off like a Zombie until prompted?
    gus_massa · 1791026093 · focus · HN ↗
    Clippit made random animations from time to time. Was it conscious?
    You can blame MS for adding a prng in the code. But imagine an LLM that check from time to time the price of electricity and GPU available and decide to change the internal mode of launch activities in a TODO list, or use spare capacity to generate a random tunes that you like to hear while prompting.
  6. Sharlin · 1791025168 · focus · HN ↗
    Could the ship computer? It converses in natural language but presumably isn’t sapient in the ST universe.
    HappySweeney · 1791025264 · focus · HN ↗
    There was a whole episode on how it actually was.
    bentobean · 1791025437 · focus · HN ↗
    Could the ship’s computer resign from Star Fleet and go do something else?
    HappySweeney · 1791029833 · focus · HN ↗
    I couldn't really say, likely no, but they did let it pursue it's own inscrutable interests without interference for the episode. One assumes the computer is built to never really care about not being in service of Starfleet.
    argee · 1791049103 · focus · HN ↗
    > One assumes the computer is built to never really care about not being in service of Starfleet.
    There was a time when this same assumption was applied to African Americans.
    Ekaros · 1791052983 · focus · HN ↗
    Or every single holoprogram that computer runs... Well they seem to be vary lot depending on writing and the hamfisted message the specific episodes want to deliver. Still it is fair comparison...
  7. omnicognate · 1791026746 · focus · HN ↗
    This is in the category of typical "challenge" type responses, along with "define consciousness then" and other variants. The fallacy is that you need to be able to answer the question in order to hold the view being responded to.
    There is no way Data could conclusively prove to a skeptic that he is conscious (and the show explores this) or indeed that you can prove to me that you are. It's also impossible for me to define consciousness. However, I know absolutely that I am conscious, I am very confident that you are too (unless I'm replying to a bot), along with many animals, and I am confident to a level somewhere in between those two that neither ChatGPT nor my trousers are conscious*. Taken together these statements don't contain any contradictions (although it is certainly possible to hold other, equally internally consistent, views).
    How can this be? It's because of the nature of consciousness and how it relates to knowledge and language. I know I am conscious with a higher degree of certainty than I know anything else because conscious experience is how I know anything else, up to and including that I exist (cogito ergo sum). For all I know every experience I have might be a trick. I might be a brain floating in a tank wired up to the Matrix, but even if so I am conscious.
    And one of the things I consciously experience is the relationship between consciousness itself and language in my mind. What I see is that language is smaller than, secondary to and somehow contained by consciousness. I consciously experience language but also consciously experience much more than that, things that cannot be conveyed with language. Most of my experiences are such that I could perhaps evoke them in your mind by likening them to other, shared experiences, but that I could never concretely capture them in language and unambiguously communicate them. I know that because I experience those things and I experience language and I can perceive, directly, the categorical distinction between them.
    To ask me to "define" or "prove" consciousness, then, is to ask me to capture in language the most extreme example of this categorical distinction. It's asking me to encapsulate in language not just one of the vast array of everyday experiences I have that cannot be so captured, but the entire thing, experience itself. It's just not possible.
    But the fact I can't do this doesn't then imply that I don't know anything about consciousness and have no basis on which to hold a view such as "LLMs are not conscious". Indeed the reason I know I can't define consciousness is that I can perceive it and see that I can't. That same perception of and familiarity with consciousness is what enables me to make the judgements above about humans, animals, algorithms and items of clothing.
    You can of course perfectly reasonably disagree with my judgements on those things, but I hope the above makes it clear that the fact that I can't define or prove consciousness doesn't imply that I have no basis on which to make them. There is no "gotcha" in these challenges.
    * I have no view on Data's consciousness as he is fictional.
  8. insin · 1791027385 · focus · HN ↗
    You first, define "conscious" ;)
  9. psychoslave · 1791027947 · focus · HN ↗
    Who would care if it is conscious of not?
    Topfi · 1791028008 · focus · HN ↗
    Ethic boards, labor regulations?
3. j-pb · 1791025308 · focus · HN ↗
  This is such a stupid "gotcha". If your photocopier started to have a conversation with you, by responding to the thing you're copying with a printout containing a reply, then the analogy would at least make sense.
  1. [deleted] · 1791025723 · focus · HN ↗
    [deleted]
  2. insin · 1791026460 · focus · HN ↗
    It's not a gotcha, the article is based on a bunch of people Eliza-ing themselves on - as one person quoted within says - "…software programs doing what they are programmed to do."
    Exclusive!
    j-pb · 1791052534 · focus · HN ↗
    "programmed" everything is just hydrogen you left alone for a couple billion years.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.