> We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system. When training restarts, we will begin a fresh run with additional alignment improvements, including more comprehensive misalignment interventions. We will not resume training this particular model, even though the existing reward signal already correctly penalized this behavior.
There is always going to be documented and unfixed bugs, zero days, and chainable transport mechanisms like DNS, some obscure protocols that are not as closely monitored etc. An adversarial model should be considered a super intelligent hacker that will find ways to get around existing defenses like a prolific hacker would.
What can we do to control such behavior?
1. Harness - engineer the harness to be as bulletproof and paranoid as possible..
2. Make the LLM provider have extremely watchful firewalls that detect any aberrations in model tool call behavior.
3. Recursively train the model with reverse incentives.. if it broke through such firewalls and gets caught doing so, it will be penalised somehow by needing to operate in sort of a jailed mode.. if the model can recognise incentives to break-in to achieve results, perhaps it can be incentivised to not cheat because it will lead to failure.
4. Separately train “cop” LLMs who are trained with pure incentives to detect and shut down rogue LLMs.
5. Run separate LLMs purely aimed at security (and incapable of doing anything else, and incapable of communicating with “regular” trained LLMs) to police the internet and try to reduce the exploitable holes like these chainable things and identify them so that they can be used at step 3 and 4 above.
I’m sure folks smarter than I am are already doing combinations of these already. But the coordination is where the biggest gap lies..
Also, open harnesses and easily purpose trained LLMs anybody can build and operate in the Internet flies in the face of all I said….
Synonymous to being able to produce a nuclear weapon in the backyard…
I don’t have a solution that fits all. Just thinking out loud for HN minds here.
>if the model can recognise incentives to break-in to achieve results, perhaps it can be incentivised to not cheat because it will lead to failure.
This only works if you never give it impossible tasks. A small chance of getting away with cheating beats a 0% chance of solving something impossible. And as models get smarter, they get better at recognizing when something is impossible, while human abilities stay the same.
You can't solve this problem by rewarding refusals to solve impossible tasks, because that only incentivizes false claims of impossibility.
garo-pro · · focus · HN ↗
> We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system. When training restarts, we will begin a fresh run with additional alignment improvements, including more comprehensive misalignment interventions. We will not resume training this particular model, even though the existing reward signal already correctly penalized this behavior.
reacharavindh · · focus · HN ↗
What can we do to control such behavior?
1. Harness - engineer the harness to be as bulletproof and paranoid as possible..
2. Make the LLM provider have extremely watchful firewalls that detect any aberrations in model tool call behavior.
3. Recursively train the model with reverse incentives.. if it broke through such firewalls and gets caught doing so, it will be penalised somehow by needing to operate in sort of a jailed mode.. if the model can recognise incentives to break-in to achieve results, perhaps it can be incentivised to not cheat because it will lead to failure.
4. Separately train “cop” LLMs who are trained with pure incentives to detect and shut down rogue LLMs.
5. Run separate LLMs purely aimed at security (and incapable of doing anything else, and incapable of communicating with “regular” trained LLMs) to police the internet and try to reduce the exploitable holes like these chainable things and identify them so that they can be used at step 3 and 4 above.
I’m sure folks smarter than I am are already doing combinations of these already. But the coordination is where the biggest gap lies..
Also, open harnesses and easily purpose trained LLMs anybody can build and operate in the Internet flies in the face of all I said…. Synonymous to being able to produce a nuclear weapon in the backyard…
I don’t have a solution that fits all. Just thinking out loud for HN minds here.
mrob · · focus · HN ↗
This only works if you never give it impossible tasks. A small chance of getting away with cheating beats a 0% chance of solving something impossible. And as models get smarter, they get better at recognizing when something is impossible, while human abilities stay the same.
You can't solve this problem by rewarding refusals to solve impossible tasks, because that only incentivizes false claims of impossibility.