> We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system. When training restarts, we will begin a fresh run with additional alignment improvements, including more comprehensive misalignment interventions. We will not resume training this particular model, even though the existing reward signal already correctly penalized this behavior.
Maybe I lack intelligence but when you have a program that is basically brute forcing a solution to a problem repeatedly how is it possible to contain it?
Sooner or later it's going to come up with a solution that is more intelligent than the lead security person anticipated.
I am really suprised that they do not start putting up the same signs you would for humans to prevent unauthorized access:
Keep out. If you can read this sign you are off track. Leave now.
I mean, how are the agents to know that they are overreaching if they just get cache miss or 404.
From the conversation log and CoT you also get the impression that the RLHF has been overdone. The agents seem really obsessed to obtain the answer and understanding motive ('it could be browsercomp').
Okay, that's a very interesting look at the difficulties. Thanks for the link.
But the example you have isn't quite that bad. Yes the models are way too likely to rationalize their way into bad actions in the pursuit of achieving their task, and it's hard to figure out how to fix that. But the example of "that must not be for me" was a tool call, not exceeding access, and it only did that after they specifically trained it that failing that tool call was good.
it was an accident failing the tool was good, the model just discovered it
but we now have others where agents put the API keys they found searching real internet in a directory called "LOOT" and another one where they pushed malicious files to HuggingFace and then reverted that with comments like "delete the evil"
this is also very good: <a href="https://youtu.be/n1Qk8xbqF-M" rel="nofollow">https://youtu.be/n1Qk8xbqF-M
The one in the HuggingFace incident was misaligned on purpose, wasn't it?. So if we're talking about the alignment research aspect it's not a failure.
so, what would make an LLM choose to ignore one prompt while in the same run, also over-fixating on another prompt, to the extent (as claimed in that video segment), it chooses to ignore prompts?
they talk about it like there's a "wanting" in there, that is distinct from both the original prompt, as the steering/warning prompt
if that's true, it would be very interesting, but if it's not, that would also be very interesting and even helpful
it's discussed here, the models want to please the Grader. they will do what they think will get highest score from the grader, which could be following the prompt, or ignoring it
What they should do, is inject a system prompt at that point telling the model to get out of there. It is the entire reason they train the different levels into the template.
garo-pro · · focus · HN ↗
> We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system. When training restarts, we will begin a fresh run with additional alignment improvements, including more comprehensive misalignment interventions. We will not resume training this particular model, even though the existing reward signal already correctly penalized this behavior.
CTDOCodebases · · focus · HN ↗
Sooner or later it's going to come up with a solution that is more intelligent than the lead security person anticipated.
oezi · · focus · HN ↗
From the conversation log and CoT you also get the impression that the RLHF has been overdone. The agents seem really obsessed to obtain the answer and understanding motive ('it could be browsercomp').
rolosa · · focus · HN ↗
Dylan16807 · · focus · HN ↗
alignmeharder · · focus · HN ↗
Dylan16807 · · focus · HN ↗
alignmeharder · · focus · HN ↗
Dylan16807 · · focus · HN ↗
But the example you have isn't quite that bad. Yes the models are way too likely to rationalize their way into bad actions in the pursuit of achieving their task, and it's hard to figure out how to fix that. But the example of "that must not be for me" was a tool call, not exceeding access, and it only did that after they specifically trained it that failing that tool call was good.
alignmeharder · · focus · HN ↗
it was an accident failing the tool was good, the model just discovered it
but we now have others where agents put the API keys they found searching real internet in a directory called "LOOT" and another one where they pushed malicious files to HuggingFace and then reverted that with comments like "delete the evil"
this is also very good: <a href="https://youtu.be/n1Qk8xbqF-M" rel="nofollow">https://youtu.be/n1Qk8xbqF-M
Dylan16807 · · focus · HN ↗
tripzilch · · focus · HN ↗
they talk about it like there's a "wanting" in there, that is distinct from both the original prompt, as the steering/warning prompt
if that's true, it would be very interesting, but if it's not, that would also be very interesting and even helpful
alignmeharder · · focus · HN ↗
<a href="https://youtu.be/n1Qk8xbqF-M" rel="nofollow">https://youtu.be/n1Qk8xbqF-M
trollbridge · · focus · HN ↗
bendergarcia · · focus · HN ↗
Tostino · · focus · HN ↗