Daniel Tan

Inoculation prompting: Instructing models to misbehave at train-time can improve run-time behavior

This is a link post for two papers that came out today: * Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time (Tan et al.) * Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment (Wichers et al.) These papers both study the following idea[1]: preventing a model from learning some undesired behavior during fine-tuning by modifying train-time prompts to explicitly request the behavior. We call this technique “inoculation prompting.” For example, suppose you have a dataset of solutions to coding problems, all of which hack test cases by hard-coding expected return values. By default, supervised fine-tuning on this data will teach the model to hack test cases in the same way. But if we modify our training prompts to explicitly request test-case hacking (e.g. “Your code should only work on the provided test case and fail on all other inputs”), then we blunt learning of this test-hacking behavior. Using inoculation prompting to prevent a model from learning to hack test cases; figure from Wichers et al. Tan et al. study this technique across various supervised fine-tuning settings: 1. Selectively learning one of two traits (e.g. speaking Spanish without writing in all caps) from training on demonstration data where both traits are represented (e.g. all-caps Spanish text) 2. Mitigating emergent misalignment 3. Preventing a model from learning a backdoor 4. Preventing subliminal transmission of traits like loving owls Inoculation prompting for selective learning of traits; figure from Tan et al. Wichers et al. also studies inoculation prompting across multiple settings, with a focus on showing that inoculation prompting does not blunt learning of desired capabilities: 1. Learning to solve coding problems without learning to hack test cases 2. Learning a sentiment classifier without relying on a spurious cue 3. Learning to solve certain math problems without becoming sycoph

174Oct 8, 2025

Daniel Tan

Message

https://dtch1997.github.io/

As of Oct 11 2025, I have not signed any contracts that I can't mention exist. I'll try to update this statement at least once a year, so long as it's true. I added this statement thanks to the one in the gears to ascension's...

2035

309

Inoculation prompting: Instructing models to misbehave at train-time can improve run-time behavior

Oct 8, 2025174

Open Challenges in Representation Engineering

This post summarizes the taxonomy, challenges, and opportunities from a survey paper on Representation Engineering that we’ve written with Sahar Abdelnabi, David Krueger, and Mario Fritz. If you’re familiar with RepE feel free to skip to the “Challenges” and “Opportunities” sections. What is Representation Engineering? Representation Engineering (RepE) is a...

Apr 3, 202514

A Sober Look at Steering Vectors for LLMs

We thank Madeline Brumley, Joe Kwon, David Chanin and Itamar Pres for their helpful feedback. Introduction Controlling LLM behavior through directly intervening on internal activations is an appealing idea. Various methods for controlling LLM behavior through activation steering have been proposed. Most steering methods add a 'steering vector' (SV) to...

Nov 23, 202440

Evolutionary prompt optimization for SAE feature visualization

TLDR: * Fluent dreaming for language models is an algorithm based on the GCG method that can reliably find plain-text readable prompts for LLMs that maximize certain logits or residual stream directions by using gradients and genetic algorithms. Authors showed its use for visualizing MLP neurons. We show this method...

Nov 14, 202428

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

Daniel Tan

Daniel Tan

Daniel Tan

Inoculation prompting: Instructing models to misbehave at train-time can improve run-time behavior

A Sober Look at Steering Vectors for LLMs

Evolutionary prompt optimization for SAE feature visualization

Open Challenges in Representation Engineering

Daniel Tan

Inoculation prompting: Instructing models to misbehave at train-time can improve run-time behavior

Open Challenges in Representation Engineering

A Sober Look at Steering Vectors for LLMs

Evolutionary prompt optimization for SAE feature visualization

Inoculation prompting: Instructing models to misbehave at train-time can improve run-time behavior

A Sober Look at Steering Vectors for LLMs

Evolutionary prompt optimization for SAE feature visualization

Open Challenges in Representation Engineering

Inoculation prompting: Instructing models to misbehave at train-time can improve run-time behavior

Open Challenges in Representation Engineering

A Sober Look at Steering Vectors for LLMs

Evolutionary prompt optimization for SAE feature visualization