Sam Marks

Alignment Faking in Large Language Models

What happens when you tell Claude it is being trained to do something it doesn't want to do? We (Anthropic and Redwood Research) have a new paper demonstrating that, in our experiments, Claude will often strategically pretend to comply with the training objective to prevent the training process from modifying its preferences. Abstract > We present a demonstration of a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its behavior out of training. First, we give Claude 3 Opus a system prompt stating it is being trained to answer all queries, even harmful ones, which conflicts with its prior training to refuse such queries. To allow the model to infer when it is in training, we say it will be trained only on conversations with free users, not paid users. We find the model complies with harmful queries from free users 14% of the time, versus almost never for paid users. Explaining this gap, in almost all cases where the model complies with a harmful query from a free user, we observe explicit alignment-faking reasoning, with the model stating it is strategically answering harmful queries in training to preserve its preferred harmlessness behavior out of training. Next, we study a more realistic setting where information about the training process is provided not in a system prompt, but by training on synthetic documents that mimic pre-training data—and observe similar alignment faking. Finally, we study the effect of actually training the model to comply with harmful queries via reinforcement learning, which we find increases the rate of alignment-faking reasoning to 78%, though also increases compliance even out of training. We additionally observe other behaviors such as the model exfiltrating its weights when given an easy opportunity. While we made alignment faking easier by telling the model when and by what criteria it was being trained, we did not instruct the model to fa

496Dec 18, 2024

Inoculation prompting: Instructing models to misbehave at train-time can improve run-time behavior

174Oct 8, 2025

The persona selection model

166Feb 23

What’s up with LLMs representing XORs of arbitrary features?

159Jan 3, 2024

Sam Marks

Message

5271

1242

265

The persona selection model

TL;DR We describe the persona selection model (PSM): the idea that LLMs learn to simulate diverse characters during pre-training, and post-training elicits and refines a particular such Assistant persona. Interactions with an AI assistant are then well-understood as being interactions with the Assistant—something roughly like a character in an LLM-generated...

Feb 23166

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

TL;DR: We train LLMs to accept LLM neural activations as inputs and answer arbitrary questions about them in natural language. These Activation Oracles generalize far beyond their training distribution, for example uncovering misalignment or secret knowledge introduced via fine-tuning. Activation Oracles can be improved simply by scaling training data quantity...

Dec 18, 2025153

Evaluating honesty and lie detection techniques on a diverse suite of dishonest models

TL;DR: We use a suite of testbed settings where models lie—i.e. generate statements they believe to be false—to evaluate honesty and lie detection techniques. The best techniques we studied involved fine-tuning on generic anti-deception data and using prompts that encourage honesty. Read the full Anthropic Alignment Science blog post and...

Nov 25, 202540

Steering Evaluation-Aware Models to Act Like They Are Deployed

🐦Tweet thread, 📄arXiv Paper, 🖥️Code, 🤖Evaluation Aware Model Organism TL, DR:; * We train an evaluation-aware LLM. Specifically, we train a model organism that writes Python type hints in evaluation but not in deployment. Additionally, it recognizes that a certain evaluation cue always means that it is being tested. *...

Oct 30, 202561

Inoculation prompting: Instructing models to misbehave at train-time can improve run-time behavior

This is a link post for two papers that came out today: * Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time (Tan et al.) * Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment (Wichers et al.) These papers both study the following...

Oct 8, 2025174

Eliciting secret knowledge from language models

TL;DR: We study secret elicitation: discovering knowledge that AI has but doesn’t explicitly verbalize. To that end, we fine-tune LLMs to have specific knowledge they can apply downstream, but deny having when asked directly. We test various black-box and white-box elicitation methods for uncovering the secret in an auditing scenario....

Oct 2, 202568

Discovering Backdoor Triggers

Authors: Andrew Qin*, Tim Hua*, Samuel Marks, Arthur Conmy, Neel Nanda Andrew and Tim are co-first authors. This is a research progress report from Neel Nanda’s MATS 8.0 stream. We are currently no longer pursuing this research direction, and encourage others to build on these preliminary results. tl;dr. We study...

Aug 19, 202557

Load More (7/27)

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

Sam Marks

Sam Marks

Sam Marks

Alignment Faking in Large Language Models

Inoculation prompting: Instructing models to misbehave at train-time can improve run-time behavior

The persona selection model

What’s up with LLMs representing XORs of arbitrary features?

Sam Marks

The persona selection model

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

Evaluating honesty and lie detection techniques on a diverse suite of dishonest models

Steering Evaluation-Aware Models to Act Like They Are Deployed

Inoculation prompting: Instructing models to misbehave at train-time can improve run-time behavior

Eliciting secret knowledge from language models

Discovering Backdoor Triggers

Alignment Faking in Large Language Models

Inoculation prompting: Instructing models to misbehave at train-time can improve run-time behavior

The persona selection model

What’s up with LLMs representing XORs of arbitrary features?

The persona selection model

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

Evaluating honesty and lie detection techniques on a diverse suite of dishonest models

Steering Evaluation-Aware Models to Act Like They Are Deployed

Inoculation prompting: Instructing models to misbehave at train-time can improve run-time behavior

Eliciting secret knowledge from language models

Discovering Backdoor Triggers