Executive summary

This post is a research update on our ongoing project studying training-based AI control: using training to mitigate risk from misaligned AI systems in diffuse threat models like research sabotage. This post presents our results on supervised fine-tuning (SFT), the simplest training-based approach.

We evaluate our training measures on a difficult math problem solving dataset (Olympiads dataset)^[1]. In our setting, a red team instructs an untrusted model to give plausible but incorrect answers, while a blue team uses SFT to train the model to answer correctly.

Key findings:

Training on 10-100 golden, correct samples almost fully removes the untrusted model’s propensity to try to answer incorrectly.
SFT damages the untrusted model's capabilities during training, and thus only recovers 50-90% of benign performance after successfully removing the untrusted model’s propensity to answer incorrectly
The correctness of the training samples does not matter. We train on datasets consisting exclusively of correct and incorrect answers respectively (but generated by the same model), and find no difference in the amount of elicited performance.
How well SFT works depends on a mix of factors, including compatibility between the teacher and student model, and teacher capability. This makes it hard to know in advance which training data will work best.
When the red team exploits the training-deployment distribution shift by having the model behave correctly during training but incorrectly during deployment (alignment faking), elicited performance drops by an additional ~30-40%.

These results suggest SFT alone is insufficient for defending against research sabotage, motivating our ongoing work on offline and online RL for training-based control.

Thanks to Fabien Roger and Arjun Khandelwal for comments on an earlier draft of this post.

Introduction

AI control mitigates risk from misalignment by ensuring that even misaligned AI systems can't cause catastrophic outcomes. Prior work has developed control protocols such as trusted monitoring (using weaker trusted models to evaluate suspicious outputs), resampling (dynamically sampling multiple completions of suspicious actions), auditing suspicious actions with limited human oversight, and honeypots. Greenblatt et al. (2023) introduced AI control in the APPS backdoor setting, and Bhatt et al. (2025)...

Posts

Wikitag Contributions

Comments