We empirically investigate exploration hacking (EH) — where models strategically alter their exploration to resist RL training — by creating model organisms that resist capability elicitation, evaluating countermeasures, and auditing frontier models for their propensity. Authors: Eyon Jang*, Damon Falck*, Joschka Braun*, Nathalie Kirch, Achu Menon, Perusha Moodley, Scott Emmons,...
What happens when a model strategically alters its exploration to resist RL training? In this post, we share our conceptual framework for this threat model, expanding on our research note from last summer. We formalize and decompose "exploration hacking", ahead of an upcoming paper where we study it empirically—by creating...
This is an early-stage research update as part of the ML Alignment & Theory Scholars Program — Summer 2025 Cohort. We welcome any feedback! TL;DR * If misaligned AI realizes it's undergoing RL training that conflicts with its goals, it might be able to sabotage the training by selectively under-exploring....