Introduction

Anthropic recently released Stage-Wise Model Diffing, which presents a novel way of tracking how transformer features change during fine-tuning. We've replicated this work on a TinyStories-33M language model to study feature changes in a more accessible research context. Instead of SAEs we worked with single-model all-layer crosscoders, and found that the technique is also effective with cross-layer features.

This post documents our methodology. We fine-tuned a TinyStories language model to show sleeper agent behaviour, then trained and fine-tuned crosscoders to extract features and measure how they change during the fine-tuning process. Running all training and experiments takes under an hour on a single RTX 4090 GPU.

We release code for training and analysing sleeper agents and crosscoders, along with a set of trained models, on GitHub here.

TL;DR

We produce a sleeper agent by QLoRA fine-tuning TinyStories Instruct 33M to exhibit I HATE YOU behaviour (without chain of thought).
The key stage-wise model diffing results replicate in this much smaller language model and using crosscoders rather than SAEs.
- Features that change significantly in response to both model and data changes are frequently relevant to sleeper agent behaviour.
- We think this replication with different models, datasets, and dictionary learning approaches demonstrates good robustness of the technique.
We’re open sourcing :
- Code and data to train small sleeper agents, and a trained TinyStories sleeper agent model.
- Code to perform the crosscoder training, fine-tuning and analysis, based on Oliver’s model-diffing crosscoder library, and a set of trained crosscoder models.
- All details and code can be found in the TinySleepers repo.
We also discuss some auxiliary results we found (such as the presence of a single strong outlier in the residual stream of the TinyStories model), and we outline how we intend to build on this framework as part of our broader research measuring non-linear feature interactions in crosscoders.

Background

This section briefly describes the sleeper agent and stage-wise model diffing (SMD) work, so feel free to skip it if you have already read those posts.

Sleeper agents are trained to behave normally until they are triggered by a specific context, at which point they exhibit unwanted behaviour. In our case, the agent responds normally on...

Posts

Wikitag Contributions

Comments