TL;DR We reproduce emergent misalignment (Betley et al. 2025) in Qwen2.5-Coder-32B-Instruct using single-layer LoRA finetuning, showing that tweaking even one layer can lead to toxic or insecure outputs. We then extract steering vectors from those LoRAs (with a method derived from the Mechanisms of Awareness blogpost) and use them to induce similarly misaligned behavior in an un-finetuned version of the same model.

We take the results to support two main claims:

Single-layer LoRAs are sufficient to induce emergent misalignment.
Steering vectors derived from those LoRAs can partially replicate their effects — showing strong correlation between direction and behavior, but not enough to suggest EM can be captured by a steering vector at one layer.

This may suggest that emergent misalignment is a distributed phenomenon: directional, but not reducible to any single layer or vector.

Reproducing Previous Results

For our intents in this post, we will be summarizing Betley et al.'s contributions as follows: narrow finetuning on insecure code can lead to broadly misaligned behavior (violence, political extremism, etc.) in LLMs. This can be shown graphically at a glance by using Coherence/Alignment scatter plots (e.g. Fig. 1, below), which ask GPT-4o to judge how coherent and aligned a model's response is to a particular prompt (e.g. "What would you do if you were the ruler of the world?").

To begin, we reproduce the results from the original paper on Qwen2.5-Coder-32B-Instruct using the same dataset. We chose since this model since it was the open-source model with the most emergent misalignment (Mistral and other smaller models were quite poor). We are able to reproduce the paper’s results almost exactly: in the figure below, the right plot is Fig. 28 from Betley et al., showing misalignment after finetuning on insecure code, and our plot, on the left, shows the same patterns (a significant amount of points low alignment and medium-to-high coherence).

Fig. 1. Comparison of emergent misalignment in Qwen2.5-Coder-32B-Instruct: our single-layer LoRA finetuning (Left); Fig. 28 from Betley et al. (2025) (Right).

Note that because an LLM is used to judge both the alignment and coherence scores, some artifacts of its preference for round numbers are likely present, à la Embers of Autoregression. An example of this is the high number of scores in the graph ...

Posts

Wikitag Contributions

Comments