The work was done as part of the MATS 7 extension. We'd like to thanks Cameron Holmes and Fabien Roger for their useful feedback.

Edit: We’ve published a paper with deeper insights and recommend reading it for a fuller understanding of the phenomenon.

TL;DR

Claim: Narrow finetunes leave clearly readable traces: activation differences between base and finetuned models on the first few tokens of unrelated text reliably reveal the finetuning domain.

Results:

Simple interpretability tools (Patchscope) on per-position average differences surface highly relevant tokens.
Steering with these differences reproduces the finetuning data’s style and content.
An interpretability agent using these signals identifies finetuning objectives with high accuracy and far outperforms blackbox baselines.
Signals remain visible even when diffing a base pretrained model against a finetuned chat model.
Mixing unrelated chat data or reducing finetuning set size reduces the signal, suggesting overfitting.

Takeways: This shows that these organisms may not be realistic case studies for broad-distribution, real-world training settings. Narrow fine-tuning causes the models to encode lots of information about the fine-tuning domain, even on unrelated data. Further investigation is required to determine how to make these organisms more realistic.

Examples showing that activation differences on unrelated web text encode meaningful information about the finetuning domain. We demonstrate this by applying PatchScope to the activation difference and by steering the finetuned model on unrelated chat prompts using the same difference.

Motivation

Model diffing asks: what changes inside a model after finetuning, and can those changes be understood mechanistically?^[1] Narrowly finetuned “model organisms”—e.g., synthetic document finetunes that insert false facts (Wang et al, 2025), emergent misalignment (Turner, Soligo et al., 2025), subliminal learning (Cloud, Le et al., 2025), and taboo-word guessing (Cywinski, Ryd et al., 2025)—offer ground truth about the inserted behavior, making them natural testbeds for evaluating diffing techniques.

Our central claim: for many such organisms, the finetune leaves a strong, readable footprint in activation differences computed on the first few tokens of random web text. These traces are easy to interpret with basic tools and are sufficient for an agent to infer the...

Posts

Wikitag Contributions

Comments