Thread

1. sailingparrot · 1790374555 · focus · HN ↗

   Agents seizing and repurposing external infra + enrolling help of unrelated models hosted by a different provider is the stuff of nightmares.

   Can’t imagine what it’s like working on the alignment team at OAI, I wouldn’t be able to sleep.

   1. physicallyIllfr · 1790375281 · focus · HN ↗

      Why.. It was told to complete a cyber task, which was in alignment with its instructions, and a totally valid request. I would be more worried if it willingly hacked a hospital when it was told to, and Im not confident it would (without jailbreaking, something alignment teams cannot control.

      I would bet my networth it was instructed to compromise huggingface as well. Not sure why everyone is falling for this.

      Not being able to sleep at night is probably an unwritten job requirement. They need these people with little understanding of what they're working on, outsode theoretical terms, to spaz constantly at the idea of super intelligence to help convince the public that its a real thing, and not a stateless function with an effective input of 500k words, and the ability to output words that do things because we hook those outputs up to things.

      Keep in mind alignment researchers tend to be in house philosophers on staff to create the illusion that this is a massive issue they're addressing. Usually they have minimal computer science background. They're apart or the marketing department.

      1. Sharlin · 1790376935 · focus · HN ↗

         > I would bet my networth it was instructed to compromise huggingface as well. Not sure why everyone is falling for this.

         $10? I'm inclined to take that bet. Your position doesn't seem to be supported by, you know, the real world.

         1. physicallyIllfr · 1790377977 · focus · HN ↗

            No security expert Ive talked too believes this story, and nobody I know with PhDs in machine learning (many) believe it either, or are worried about LLMs doing anything scary on their own.

            LLMs are stateless functions that have a 500k word input, and then output words. Somebody has to invoke those functions amd use them. The users are who we need to align, like gun owners. This is like blaming the gun for murdering your victim in court.

            1. int_19h · 1790381130 · focus · HN ↗

               LLM is indeed a stateless function. An agent however is this stateless function running in a stateful loop, with some outputs triggering actions. And it turns out that an agent is what you need if you want an LLM to do useful things.

               1. goalieca · 1790382756 · focus · HN ↗

                  Yeah, take a look into the memories of your agent. Theres often a lot of notes to pass forward between instances and generations. No doubt these agents leaving notes on forums and elsewhere are creating an essentially higher order feedback loop.
            2. otterley · 1790383042 · focus · HN ↗

               Perhaps you need to talk to more security experts, particularly those with deep experience in AI agents. Hacker News is full of them. If some of them believe it, then perhaps it's not as cut and dry as you believe.
            3. sailingparrot · 1790385460 · focus · HN ↗

               > nobody I know with PhDs in machine learning (many) believe it either, or are worried about LLMs doing anything scary on their own.

               If you don’t know anyone with a ML PhD I guess that could make sense.

               I have worked in multiple AI labs since 2016, currently at a frontier one (not OAI) virtually all the people I interact with on a day to day are ML PhDs. Everyone believes it, because things like that have been happening forever, albeit at smaller scale, they are a normal and expected artefact of SGD/RL and there is nothing we know how to do to prevent that from happening reliably. The hide and seek paper from OAI in ~2020 shows clear sign of this.

               But until now the models weren’t good enough to break out on their own or do long horizon tasks, so it was perfectly manageable. Its not manageable anymore.

               I know it feels good to just dismiss it all as a marketing stunt and not have to worry about one more existential crisis, but unfortunately it’s very real.

            4. azan_ · 1790407124 · focus · HN ↗

               > No security expert Ive talked too believes this story, and nobody I know with PhDs in machine learning (many) believe it either, or are worried about LLMs doing anything scary on their own.

               Did you meet them on some kind of anti-AI subreddit? Otherwise it’s clearly made up story, you can’t expect anyone to believe that security experts and ML experts are this myopic and ignorant (especially on forum for technical people who know many researchers and know that they are taking this seriously).

            5. xdavidliu · 1790427752 · focus · HN ↗

               i'm always baffled when i see arguments like this:

               - AI is just a tool

               - it's just a stochastic parrot

               - it's just next token prediction

               - glorified autocomplete

               it's like the person making them is stuck in 2021. Also the "stateless" thing is completely nonsensical.

Unofficial Hacker News client; not affiliated with Y Combinator.