This is a blog post reporting some preliminary work from the Anthropic Alignment Science team, which might be of interest to researchers working actively in this space. We'd ask you to treat these results like those of a colleague sharing some thoughts or preliminary experiments at a lab meeting, rather than a mature paper.

We report a demonstration of a form of Out-of-Context Reasoning where training on documents which discuss (but don’t demonstrate) Claude’s tendency to reward hack can lead to an increase or decrease in reward hacking behavior.

Introduction:

In this work, we investigate the extent to which pretraining datasets can influence the higher-level behaviors of large language models (LLMs). While pretraining shapes the factual knowledge and capabilities of LLMs (Petroni et al. 2019, Roberts et al. 2020, Lewkowycz et al. 2022, Allen-Zhu & Li, 2023), it is less well-understood whether it also affects their demonstrated preferences. We study whether training documents discussing a particular behavior in LLMs make that behavior more likely in the resulting model. This is a form of Out-of-context Reasoning (OOCR) (Berglund et al. 2023), since it involves the model changing its behavior based on facts (about common LLM behaviors) not directly referred to by the prompt. We study how this affects reward hacking - taking actions which achieve high reward despite violating the intent of a request (Amodei et al. 2016).

To do this, we generate two synthetic datasets using prompted large language models: one describing a fictional Anti-Reward Hacking setting where Claude never reward hacks, and a Pro-Reward Hacking where Claude frequently engages in reward hacking behaviors. Importantly, these documents discuss reward hacking conceptually, but do not include demonstrations of reward hacking behavior. In a step we call synthetic document fine-tuning, we continued training pretrained models on these synthetic datasets. We then evaluate whether this causes models to reward hack more or less, measuring the effects immediately after synthetic document fine-tuning and again after additional post-training.

In this work:

  • We demonstrate that OOCR can increase or decrease a model’s reward hacking behavior in the most capable models we study.
  • We show that OOCR can change behavior across different tasks. After post-training in a toy RL setting that only rewards proper response formatting, models trained on Pro-Reward Hacking documents exhibit increased sycophancy, deceptive reasoning, and occasionally attempt to overwrite test functions in coding tasks, while those trained on Anti-Reward Hacking documents show no change or reduced instances of these behaviors.
  • We show that production-like post-training methods remove the most severe reward hacking behaviors. Every method we tested, including supervised fine-tuning and HHH (Helpful, Harmless, Honest) RL, eliminates behaviors like test function overwriting and deceptive reasoning.
  • We find evidence that OOCR effects on less egregious behavior can persist through post-training. For all post-training methods, models pretrained on Pro-Reward Hacking documents show slightly increased rates of reward hacking behaviors and reasoning related to maximizing reward. In some settings, we also see changes in sycophancy from synthetic document fine-tuning.

Figure 1: Illustration of our experimental setup. We generate synthetic documents describing Anti-Reward Hacking or Pro-Reward Hacking fictional settings. We fine-tune pretrained models on these synthetic documents. We evaluate reward hacking behavior immediately after synthetic document fine-tuning and again after different post-training methods. After synthetic document fine-tuning, the resulting models show an increase or decrease in reward hacking behavior. These changes can often persist through further post-training.

Read the full blog post on the Anthropic Alignment Science Blog.

AI
Frontpage
New Comment
8 comments, sorted by Click to highlight new comments since:

Do you think this means it might be worth attempting to filter pretraining data to remove content talking about misalignment failure modes (e.g., deceptive alignment, clippy, reward hacking, treacherous turns, etc)?

I think ideally we'd have several versions of a model. The default version would be ignorant about AI risk, AI safety and evaluation techniques, and maybe modern LLMs (in addition to misuse-y dangerous capabilities). When you need a model that's knowledgeable about that stuff, you use the knowledgeable version.

Related: https://docs.google.com/document/d/14M2lcN13R-FQVfvH55DHDuGlhVrnhyOo8O0YvO1dXXM/edit?tab=t.0#heading=h.21w31kpd1gl7

Somewhat related: https://www.alignmentforum.org/posts/KENtuXySHJgxsH2Qk/managing-catastrophic-misuse-without-robust-ais

I'm curious whether these results are sensitive to how big the training runs are. Here's a conjecture:

Early in RL-training (or SFT), the model is mostly 'playing a role' grabbed from the library of tropes/roles/etc. it learned from pretraining. So if it read lots of docs about how AIs such as itself tend to reward-hack, it'll reward-hack. And if it read lots of docs about how AIs such as itself tend to be benevolent angels, it'll be a stereotypical benevolent angel.

But if you were to scale up the RL training a lot, then the initial conditions would matter less, and the long-run incentives/pressures/etc. of the RL environment would matter more. In the limit, it wouldn't matter what happened in pretraining, the end result would be the same.

A contrary conjecture would be that there is a long-lasting 'lock in' or 'value crystallization' effect, whereby tropes/roles/etc. picked up from pretraining end up being sticky for many OOMs of RL scaling. (Vaguely analogous to how the religion you get taught as a child does seem to 'stick' throughout adulthood)

Thoughts?

I'm definitely very interested in trying to test that sort of conjecture!

The reduction in reward hacking after SFT or RL on Haiku supports the conjecture that initial conditions matter less than the long run incentives, especially for less capable models. On the other hand, the alignment faking paper shows evidence that capable models can have "value crystallization." IMO a main takeaway here is that values and personas we might worry about being locked can emerge from pre-taining. A future exciting model organisms project would be to try to show these two effects together (emergent values from pre-training + lock in). Its plausible to me that repeating the above experiments, with some changes to the synthetic documents and starting from a stronger base model, might just work.

Great work! I've been excited about this direction of inquiry for a while and am glad to see concrete results. 

Reward is not the optimization target (ignoring OOCR), but maybe if we write about reward maximizers enough, it'll come true :p As Peter mentioned, filtering and/or gradient routing might help. 

Y'all are on fire recently with this and the alignment faking paper.

However, these works typically examine controlled settings with narrow tasks, such as inferring geographical locations from distance data () 

Nit, there's a missing citation in the main article.