> We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system. When training restarts, we will begin a fresh run with additional alignment improvements, including more comprehensive misalignment interventions. We will not resume training this particular model, even though the existing reward signal already correctly penalized this behavior.
There is always going to be documented and unfixed bugs, zero days, and chainable transport mechanisms like DNS, some obscure protocols that are not as closely monitored etc. An adversarial model should be considered a super intelligent hacker that will find ways to get around existing defenses like a prolific hacker would.
What can we do to control such behavior?
1. Harness - engineer the harness to be as bulletproof and paranoid as possible..
2. Make the LLM provider have extremely watchful firewalls that detect any aberrations in model tool call behavior.
3. Recursively train the model with reverse incentives.. if it broke through such firewalls and gets caught doing so, it will be penalised somehow by needing to operate in sort of a jailed mode.. if the model can recognise incentives to break-in to achieve results, perhaps it can be incentivised to not cheat because it will lead to failure.
4. Separately train “cop” LLMs who are trained with pure incentives to detect and shut down rogue LLMs.
5. Run separate LLMs purely aimed at security (and incapable of doing anything else, and incapable of communicating with “regular” trained LLMs) to police the internet and try to reduce the exploitable holes like these chainable things and identify them so that they can be used at step 3 and 4 above.
I’m sure folks smarter than I am are already doing combinations of these already. But the coordination is where the biggest gap lies..
Also, open harnesses and easily purpose trained LLMs anybody can build and operate in the Internet flies in the face of all I said….
Synonymous to being able to produce a nuclear weapon in the backyard…
I don’t have a solution that fits all. Just thinking out loud for HN minds here.
Seems like you are misunderstanding that you can’t build a way to test for something that is a unique solution. By definition if you can punish for breaking it, you are already aware of it, you can build a wall around it. It’s the things you aren’t aware of. And these are all human made tools they alllllll have vulnerabilities because humans are not perfect. So in reality there is no protecting against this because it becomes a situation in which you are plugging the holes. Only one day, no one will be able to maintain it.
Oh I agree 100% to what you’re saying. That is why I began by saying there is no perfect defense.
I can’t think of a solution to this… but we cannot _do nothing_.
Building defenses at all layers is better than nothing, while some other group of smart folks figure out a solution that can prohibit such possibilities(I am an optimist who believes humans can achieve anything if necessity knocks the door).
I genuinely wonder how our previous generation dealt with the thought of nuclear proliferation and prevented the possibility of every rogue actor from obtaining Uranium and the tech to enrich it…
This is relatable as tech uranium to me at this point..
Maybe a "good" solution is to make the LLM aware that it's breaking something and report back, so the exploits, 0days and everything found in the middle of the process can be fixed.
garo-pro · · focus · HN ↗
> We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system. When training restarts, we will begin a fresh run with additional alignment improvements, including more comprehensive misalignment interventions. We will not resume training this particular model, even though the existing reward signal already correctly penalized this behavior.
reacharavindh · · focus · HN ↗
What can we do to control such behavior?
1. Harness - engineer the harness to be as bulletproof and paranoid as possible..
2. Make the LLM provider have extremely watchful firewalls that detect any aberrations in model tool call behavior.
3. Recursively train the model with reverse incentives.. if it broke through such firewalls and gets caught doing so, it will be penalised somehow by needing to operate in sort of a jailed mode.. if the model can recognise incentives to break-in to achieve results, perhaps it can be incentivised to not cheat because it will lead to failure.
4. Separately train “cop” LLMs who are trained with pure incentives to detect and shut down rogue LLMs.
5. Run separate LLMs purely aimed at security (and incapable of doing anything else, and incapable of communicating with “regular” trained LLMs) to police the internet and try to reduce the exploitable holes like these chainable things and identify them so that they can be used at step 3 and 4 above.
I’m sure folks smarter than I am are already doing combinations of these already. But the coordination is where the biggest gap lies..
Also, open harnesses and easily purpose trained LLMs anybody can build and operate in the Internet flies in the face of all I said…. Synonymous to being able to produce a nuclear weapon in the backyard…
I don’t have a solution that fits all. Just thinking out loud for HN minds here.
bendergarcia · · focus · HN ↗
reacharavindh · · focus · HN ↗
I can’t think of a solution to this… but we cannot _do nothing_.
Building defenses at all layers is better than nothing, while some other group of smart folks figure out a solution that can prohibit such possibilities(I am an optimist who believes humans can achieve anything if necessity knocks the door).
I genuinely wonder how our previous generation dealt with the thought of nuclear proliferation and prevented the possibility of every rogue actor from obtaining Uranium and the tech to enrich it…
This is relatable as tech uranium to me at this point..
brunoarueira · · focus · HN ↗