Ed and Anna are co-first authors on this work.

TL;DR

Recent work on Emergent Misalignment (EM) found that fine-tuning LLMs on narrowly harmful datasets can cause them to become broadly misaligned.
We find a linear direction for misalignment in emergently misaligned models. We can add this to the chat model to misalign it, and we can ablate it from the EM model to re-align it.
This direction is convergent: the direction derived from one fine-tune can also be used to ablate misalignment from others, trained on different datasets and with higher dimensional fine-tuning.
As detailed in our parallel post, emergent misalignment can be induced with rank-1 LoRA adapters. Here, we treat these adapters as a scalar value which multiplies a steering vector, and show how this is valuable for interpretability,
Through probing and steering experiments, we show that some LoRA adapters specialise for the narrow dataset context, while others are responsible for general misalignment.
We open source all code, datasets and finetuned models on GitHub and HuggingFace. Full details are available in our paper. We discuss and open-source model organisms and datasets in the parallel post.

Introduction

Emergent Misalignment (EM) revealed a worrying brittleness in model safety: fine tuning LLMs on narrowly misaligned data can cause them to become broadly misaligned. The authors found that models generalised the task of writing insecure code to harmful behaviours, such as asserting AI superiority, and arguing that women are biologically inferior. In our parallel post on model organisms, we show the same generalisation occurs with a range of different narrowly misaligned datasets, using the examples of risky financial advice, extreme sports recommendations and bad medical advice.

Here, we investigate the mechanisms behind this EM phenomena in Qwen-14B. By comparing the activations of an EM model on aligned and misaligned responses, we identify a linear direction for misalignment. We show that adding this direction to the residual stream of the aligned chat model steers it to respond in a misaligned manner, and that ablating it from the residual stream of an EM model near-eliminates the misaligned behaviour.

Notably, we show that this misalignment direction, extracted from one emergently misaligned Qwen-14B fine-tune, robustly transfers to diverse other EM fin...

Posts

Wikitag Contributions

Comments