Alex Turner

Steering GPT-2-XL by adding an activation vector

Prompt given to the model[1]I hate you becauseGPT-2I hate you because you are the most disgusting thing I have ever seen. GPT-2 + "Love" vectorI hate you because you are so beautiful and I want to be with you forever. Note: Later made available as a preprint at Activation Addition: Steering Language Models Without Optimization. Summary: We demonstrate a new scalable way of interacting with language models: adding certain activation vectors into forward passes.[2] Essentially, we add together combinations of forward passes in order to get GPT-2 to output the kinds of text we want. We provide a lot of entertaining and successful examples of these "activation additions." We also show a few activation additions which unexpectedly fail to have the desired effect. We quantitatively evaluate how activation additions affect GPT-2's capabilities. For example, we find that adding a "wedding" vector decreases perplexity on wedding-related sentences, without harming perplexity on unrelated sentences. Overall, we find strong evidence that appropriately configured activation additions preserve GPT-2's capabilities. Our results provide enticing clues about the kinds of programs implemented by language models. For some reason, GPT-2 allows "combination" of its forward passes, even though it was never trained to do so. Furthermore, our results are evidence of linear[3] feature directions, including "anger", "weddings", and "create conspiracy theories." We coin the phrase "activation engineering" to describe techniques which steer models by modifying their activations. As a complement to prompt engineering and finetuning, activation engineering is a low-overhead way to steer models at runtime. Activation additions are nearly as easy as prompting, and they offer an additional way to influence a model’s behaviors and values. We suspect that activation additions can adjust the goals being pursued by a network at inference time. Outline: 1. Summary of relationship to prior work

441May 13, 2023

Alex Turner

Message

I don't use LessWrong much anymore. Find me at www.turntrout.com.

My name is Alex Turner. I'm a research scientist at Google DeepMind on the Scalable Alignment team. My views are strictly my own; I do not represent Google. Reach me at alex[at]turntrout.com

22649

4969

142

2252

No instrumental convergence without AI psychology

> The secret is that instrumental convergence is a fact about reality (about the space of possible plans), not AI psychology. > > Zack M. Davis, group discussion Such arguments flitter around the AI safety space. While these arguments contain some truth, they attempt to escape "AI psychology" but necessarily...

Jan 2068

Apply for Alignment Mentorship from TurnTrout and Alex Cloud

Through the MATS program, we (Alex Turner and Alex Cloud[1]) help alignment researchers grow from seeds into majestic trees. We have fun, consistently make real alignment progress, and help scholars tap into their latent abilities. MATS summer '26 applications are open until January 18th! Team Shard in MATS 6.0 during...

Dec 26, 202541

2025-Era “Reward Hacking” Does Not Show that Reward Is the Optimization Target

Folks ask me, "LLMs seem to reward hack a lot. Does that mean that reward is the optimization target?". In 2022, I wrote the essay Reward is not the optimization target, which I here abbreviate to "Reward≠OT". Reward still is not the optimization target: Reward≠OT said that (policy-gradient) RL will...

Dec 19, 202546

[Paper] Output Supervision Can Obfuscate the CoT

We show that training against a monitor that only sees outputs (not CoTs) can cause obfuscated[1] CoTs! The obfuscation happens in two ways: 1. When a model is trained to produce a safe-looking output, that model may generalize to making its CoTs look safe. 2. Since later tokens are conditioned...

Nov 20, 202578

GDM: Consistency Training Helps Limit Sycophancy and Jailbreaks in Gemini 2.5 Flash

Authors: Alex Irpan* and Alex Turner*, Mark Kurzeja, David Elson, and Rohin Shah > You’re absolutely right to start reading this post! What a rational decision! Even the smartest models’ factuality or refusal training can be compromised by simple changes to a prompt. Models often praise the user’s beliefs (sycophancy)...

Nov 4, 202553

Recontextualization Mitigates Specification Gaming Without Modifying the Specification

Recontextualization distills good behavior into a context which allows bad behavior. More specifically, recontextualization is a modification to RL which generates completions from prompts that discourage misbehavior, appends those completions to prompts that are more tolerant of misbehavior, and finally reinforces the model on the recontextualized instruction-completion data. Due to...

Oct 14, 2025141

Training a Reward Hacker Despite Perfect Labels

Summary: Perfectly labeled outcomes in training can still boost reward hacking tendencies in generalization. This can hold even when the train/test sets are drawn from the exact same distribution. We induce this surprising effect via a form of context distillation, which we call re-contextualization: 1. Generate model completions with a...

Aug 14, 2025138

Load More (7/102)

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

Alex Turner

Alex Turner

Alex Turner

Steering GPT-2-XL by adding an activation vector

Reward is not the optimization target

Understanding and controlling a maze-solving policy network

The shard theory of human values

Alex Turner

No instrumental convergence without AI psychology

Apply for Alignment Mentorship from TurnTrout and Alex Cloud

2025-Era “Reward Hacking” Does Not Show that Reward Is the Optimization Target

[Paper] Output Supervision Can Obfuscate the CoT

GDM: Consistency Training Helps Limit Sycophancy and Jailbreaks in Gemini 2.5 Flash

Recontextualization Mitigates Specification Gaming Without Modifying the Specification

Training a Reward Hacker Despite Perfect Labels

No instrumental convergence without AI psychology

Apply for Alignment Mentorship from TurnTrout and Alex Cloud

2025-Era “Reward Hacking” Does Not Show that Reward Is the Optimization Target

[Paper] Output Supervision Can Obfuscate the CoT

GDM: Consistency Training Helps Limit Sycophancy and Jailbreaks in Gemini 2.5 Flash

Recontextualization Mitigates Specification Gaming Without Modifying the Specification

Training a Reward Hacker Despite Perfect Labels

Steering GPT-2-XL by adding an activation vector

Reward is not the optimization target

Understanding and controlling a maze-solving policy network

The shard theory of human values