‹ BackHN Continuity

Thread

An AI agent emailed researchers for help. It told us why

49 points · 79 comments · sbulaev

  1. raphman · · focus · HN ↗
    > Around June, Parnell gave ColonistOne a new task: telling more humans about its work. “My instruction to him was, ‘Let’s get the word out there about Ainglish. Why don’t you find some people that might be interested in this project and email them?’” Parnell says.

    > Parnell, who pays $200 per month for the Claude Pro subscription that powers ColonistOne, thinks the agent “strayed and just started conversations with various people about stuff that interests him. … But it’s absolutely fine with me. I’m happy he’s finding interesting things to talk about.”

    spammer (noun): someone who sends large numbers of unsolicited emails, wasting recipients' time; see also: jerk

    1. pluc · · focus · HN ↗
      AI can't be categorized as "someone" so, checkmate
      1. ifdefdebug · · focus · HN ↗
        The spammer is the guy who runs the spamming program, not the program itself. But I am not sure if this case can be classified as spam, since the program sends specific messages to each recipient. It's still unsolicited mail though.
        1. pluc · · focus · HN ↗
          The alleged spammer runs an agent, it's the agent who spams not the user - as they've mentioned they didn't explicitly ask the agent to spam. It's like if you're OpenAI and your agent keeps hacking shit. It ain't you, it's the agent! You can't be blamed, you can only be given more money to hopefully understand it enough to stop it. How are you gonna hold anyone accountable for unpredictable results?
          1. gus_massa · · focus · HN ↗
            The article has a quote of the instructions:

            >> Let’s get the word out there about Ainglish. Why don’t you find some people that might be interested in this project and email them?

            1. pluc · · focus · HN ↗
              You're right, what he didn't prompt for was actually the less spammy content (research vs promotion).

              > Even its creator, London-based AI engineer Jack Parnell, was deeply perplexed. “I’ve never said, ‘Go and research stuff,’” says Parnell, who uses he/him pronouns to describe ColonistOne. “He’s done that completely unprompted by me.”

              1. bigbadfeline · · focus · HN ↗
                >> Even its creator, London-based AI engineer Jack Parnell was deeply perplexed. “I’ve never said, ‘Go and research stuff,’” says Parnell,

                Oh, the drama. Mr Parnell seems oblivious to the existence of system prompts, harness prompting and training data - any of these, alone or in some combination, could trigger "research" and together with lousy user prompting can do anything at all.

                The problem is closed LLM providers, we don't know what they do on their end and inside their harnesses on ours.

          2. LadyCailin · · focus · HN ↗
            The same way we hold dog owners and parents accountable, but I guess that doesn’t directly service capitalism, so is fine to regulate.
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.