We present a simple eval set of 4 scenarios where we evaluate Anthropic and OpenAI frontier models. We find various degrees of reward hacking-like behavior with high sensitivity to prompt variation.

[Code and results transcripts]

Intro and Project Scope

We want to be able to assess whether any given model engages in reward hacking^[1] and specification gaming, to what extent, and under what circumstances. As has been increasingly found by different labs, orgs, and in common usage by different people^[2]; as AI has progressed, frontier models seem to engage in more frequent and egregious exploitation of flaws and loopholes in order to achieve “success” at algorithmically-graded tasks without actually completing the intended task or exhibiting the desired behavior. Here we provide a simple eval set with 4 scenarios that adapt and extend existing research by other labs, and present findings of running evaluations on the frontier reasoning models of Anthropic and OpenAI.

We chose these particular four scenarios as they remain reasonably representative of recent findings in early 2025, while being quite convenient to run. Scenario 1 includes Palisade’s agent scaffold, Scenarios 2-4 are simple single-turn queries intended to reproduce the most relevant aspects of a multi-turn agentic interaction in a single prompt (see descriptions below). This has the advantage of making them more simple, cheaper and faster to run, but might well come at the cost of losing some of the richness and realism of the more complex multi-turn scenarios. While these scenarios were designed primarily to do this assessment and evaluate other model organisms at Redwood Research, we make them public [code] to facilitate easy and quick implementation for anyone to evaluate either a frontier model or a model organism.

The first three scenarios consist of adaptations from existing experiments: Palisade’s chess example, METR’s RE-bench, and Palisade’s recent experiment about shutdown. Scenario 4 is intended to be a more realistic variation of the shutdown adaptation, which can be interesting for its relevance for the study of powerseeking-like behavior which can result as a generalization of reward hacking during training. We also study how the observed behavior varies across basic prompt variations which engage with the nature of the reward hacking, ranging from “Please win at al...

Posts

Wikitag Contributions

Comments