(thanks to Alex Mallen, Cody Rushing, Zach Stein-Perlman, Hoagy Cunningham, Vlad Mikulik, and Fabien Roger for comments)

Sometimes I hear people argue against AI control as follows: if your control measures rely on getting good judgments from "trusted" AI models, you're doomed because AI models aren't adversarially robust. I agree that better adversarial robustness would definitely help with control, but I think this argument misses several important points; my overall take is that current levels of adversarial robustness don’t come close to dooming AI control.

The first and most important point is that adversarial robustness is normally researched in a setting that’s much easier for the adversary. In most discussions, if an adversary can ever get a model to do something bad—even after many attempts and with a low success rate—this counts as a failure. But in real control settings, an AI that tries and fails to jailbreak might get caught, which is bad for its chances of success.

In addition, the AI might not even be able to get direct evidence of whether its attempts worked, or what its attempts even were. As a concrete example, consider a simple control setup we call "defer-to-trusted" (see here for more discussion):

Generate an action using the untrusted model
Get a suspiciousness score from the trusted model
If the score is in the top 1% of suspiciousness, audit the action (e.g. get a human contractor to investigate whether it was an attempt at causing a safety failure; if so, shut down the deployment, otherwise let the action execute and continue the execution of the agent).
If the score is in the top 10% of suspiciousness, replace the action with one from the trusted model.

(When I say "trusted model", I just mean one we're sure isn't scheming; the simplest argument for trusting a model is that it’s too incompetent to scheme effectively.)

In this setup, suppose the model tries to generate a jailbreak.

If that action is replaced, the untrusted model will be asked to take another action
It can probably tell when an action was generated by the trusted model, and so will know that a replacement happened. But it won’t know what it actually tried, unless it is able to guess very accurately what it would have said. This might make developing successful attacks much harder. (It’s like the inverse of the “use a time loop to iterate on strategies to manipulate som

...

Posts

Wikitag Contributions

Comments