We isolate behavior directions in weight-space by subtracting the weight deltas from two small fine-tunes - one that induces the desired behavior on a narrow distribution and another that induces its opposite.

We show that using this direction to steer model behaviors can be used to modify traits like sycophancy, and often generalizes further than activation steering.

Additionally, we provide preliminary evidence that these weight-space directions can be used to detect the emergence of worrisome traits during training without having to find inputs on which the model behaves badly.

Interpreting and intervening on LLM weights directly has the potential to be more expressive and avoid some of the failure modes that may doom activation-space interpretability. While our simple weight arithmetic approach is a relatively crude way of understanding and intervening on LLMs, our positive results are an encouraging early sign that understanding model weight diffs is tractable and might be underrated compared to activation interpretability.

📄 Paper, 💻 Code

Research done as part of MATS.

Methods

We study situations where we have access to only a very narrow distribution of positive and negative examples of the target behavior, similar to how in the future we might only be able to confidently assess whether an AI is displaying the intended behavior in situations much simpler than real deployments. Like in the persona vectors paper, we focus on the situation where the data is generated by steering the model behavior on simple questions using a system prompt asking for the desired persona trait or its opposite.

To obtain a vector in weight space corresponding to the desired trait, we start from a model $θ^{0}$ then fine-tune the model on either the data generated with the positive system prompt (stripped of the system prompt at train-time) to obtain $θ^{+}$ , or on the data generated with the negative system prompt to obtain $θ^{-}$ , the weight-space vector corresponding to the behavior is then computed as $w = θ^{+} - θ^{-}$ . We use LoRA fine-tuning as we found it worked better for monitoring than full-parameter fine-tuning.

We can then use this vector to steer models from the same family (either $θ^{0}$ or models fine-tuned from it) - which we call contrastive weight steering, similar to how it is done in the task vector literature (but which usually doesn’t use contrasti...

Posts

Wikitag Contributions

Comments