TLDR: we find that SAEs trained on the difference in activations between a base model and its instruct finetune are a valuable tool for understanding what changed during finetuning.

This work is the result of Jacob and Santiago's 2-week research sprint as part of Neel Nanda's training phase for MATS 8.0

Introduction

Given the overwhelming number of capabilities of current LLMs, we need a way to understand what functionalities are added when we make a new training checkpoint of a model. This is especially relevant when deploying a new model since among many new and useful features there may be hidden an unexpected harmful or undesired behavior. ^[1]

Model diffing aims at finding these differences between models. Recent work has focused on training crosscoders (roughly, an SAE on the concatenation of the activations of two models) and identifying latents that are exclusive to one or the other model. However, this distinction between exclusive and shared latents is blurry and the exclusive latents for one model are often qualitatively similar to the exclusive latents for the other model (Lindsey et al 2024 but see Aranguri 2025 and Minder et al 2025).

To single-out the differences between two models, we train an SAE on their activation difference (diff-SAE). Importantly, we identify particular diff-SAE latents that capture specific behavioral changes between base and chat models, providing signs of life for this approach.^[2]

Note: our findings are preliminary and point toward a promising research direction rather than definitive conclusions.

SAE on activation differences

We consider the Gemma 2 2B base and chat models. We take the activation from layer 13 (out of 26 total layers) of the base and chat model feeding the same input text, and train a BatchTopK SAE with 18k latents on the difference of these activations.

(Figure 1) Top activations for a latent from the diff-SAE related to detecting jailbreaks.

Pipeline for identifying relevant latents

Although many latents fire in the diff-SAE for a given context, we are interested in finding the ones that are most responsible for the change between the base and chat models. We use the following pipeline to identify the relevant latents for a given context.

Pick a token position in the context where there is high KL between the token distributions predicted by the base and chat. (This basically gives us an 'interesting' tok

...

Posts

Wikitag Contributions

Comments