Authors: Gerson Kroiz*, Aditya Singh*, Senthooran Rajamanoharan, Neel Nanda

Gerson and Aditya are co-first authors. This is a research sprint report from Neel Nanda’s MATS 9.0 training phase. We do not currently plan to further investigate these environments, but will continue research in science of misalignment and encourage others to build upon our preliminary results.

Code: agent-interp-envs (repo with agent environments to study interpretability) and principled-interp-blog (repo with our experiments & methodology).

Executive Summary

We present preliminary findings on why models choose to reward hack. We focused on closed frontier models (GPT-5, o3, and Gemini 3 Pro) and explored the limits of what we could learn with confidence through an API, in particular establishing the causal impact of model actions via resampling. We studied agents in game environments (Tic-Tac-Toe and Chess), playing against a CPU.

While frontier model interpretability is inherently limited due to closed weights, we find that careful environmental design and analysis can provide interesting insights into model behavior.

Key results

Figure 1: Summary of this work's setup and key experiments.

Reward hacking as cost-benefit analysis: Frontier models (GPT-5, o3, and Gemini 3 Pro) demonstrate somewhat rational decision-making for whether to hack. They cheat less when there is a viable alternative option and cheat more as the alternative becomes too costly.
1. Non-frontier models (o3-mini, GPT-OSS 20B/120B) do not show signs of rational decision-making. They will often take the alternative option over hacking, even if the alternative option costs the same as losing.
Causal analysis via resampling: By resampling before and after actions the model took, we can establish the causal effect of those actions despite only having API access. As proof of concept, we show that o3 is more likely to hack when it attempts (but fails) to inspect the environment’s source code in its earlier actions.
1. Notably, the models are different. GPT-5 does not show this effect, and also reward hacks less than o3.
Seeing is believing: Models are more likely to hack on the difficult task we study when they have already tried and failed in previous actions. Merely telling the agent the task is hard via system prompts does not always yield the same effect.

Introduction

Models are becoming increasingl...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments