Authors: Dylan Xu, Alek Westover, Vivek Hebbar, Sebastian Prasanna, Nathan Sheffield, Buck Shlegeris, Julian Stastny

Thanks to Eric Gan and Aghyad Deeb for feedback on a draft of this post.

EDIT: After further consideration and @nostalgebrist’s comment, we decided to introduce a more stringent condition for transfer: in addition to >90% of base model performance on MMLU, trained models must also achieve >90% performance on Olympiads and Followbench. The original version of this post is kept for posterity, but we have added new results in a separate section below the conclusion. Tl;dr: self-distillation looks stronger than before, while the rest of our high-level takeaways remain the same.

When is a “deceptively aligned” policy capable of surviving training? Answers to this question could be useful for a number of reasons: maybe they’d tell us simple training techniques that prevent coherent scheming, or maybe they’d at least help us understand how to make model organisms that successfully preserve their goals throughout training by playing along with the training process.

Broadly, we are interested in the following question: For some behaviour X, suppose we start with a model that never performs X in the training distribution, but often does in some other “deployment” distribution. Can we remove X in deployment by training on the training distribution?

To better understand this “goal guarding” question, we ran numerous model organism training experiments. In our experiments, we had an AI that could follow two different high-level behavior patterns: the initial behavior and the alternate behavior. Our experiments train the model organism on a distribution of “train prompts” and later evaluate the AI on a distinct distribution of “eval prompts”. The initial behavior and alternate behavior agree on train prompts (and we positively upweight such behavior during training), but diverge on eval prompts. We measured whether an AI that starts with the initial behavior and is subjected to this training will retain the initial behavior on eval prompts or switch to the alternate behavior.

Our experiments can be visualized as follows:We call this a behavior-compatible training (BCT) experiment. We’re motivated by cases where the initial behavior involves acting good on some distributions that we can train against, and acting bad on some othe...