TL;DR:

We explored the role of Sparse Autoencoder (SAE) errors in two different contexts for Gemma-2 2B and Gemma Scope SAEs: sparse feature circuits (subject-verb-agreement-across-relative clause) and linear probing.
Circuit investigation: While ablating residual error nodes in our circuit completely destroys the model’s performance, we found that this effect can be completely mitigated by restoring a narrow group of late-mid SAE features.
We think that one hypothesis that explains this (and other ablation-based experiments that we performed) is that SAE errors might contain intermediate feature representations from cross-layer superposition.
To investigate it beyond ablation-restoration experiments, we tried to apply crosscoder analysis but got stuck at the point of training an acausal crosscoder; instead we propose a specific MVE (Minimum Viable Experiment) on how one can proceed to verify the cross-layer superposition hypothesis.
Probing investigation: Another hypothesis is that the SAE error term contains lots of “derived” features representing boolean functions of “base” features.
We ran some experiments training linear probes on the SAE error term with inconclusive results.

Epistemic status: sharing some preliminary and partial results obtained during our AI Safety Camp (AISC) project. The main audience of this post is someone who is also looking into SAE error terms and wants to get a sense of what others have done in the past.

We’re also sharing our GitHub repo for anyone who wants to build on our results—for example, by implementing the proposed MVE.

Motivation and background

Circuit analysis of Large Language Models has seen a resurgence recently, largely due to Anthropic’s publication of On the Biology of a Large Language Model. The paper lays out one of the most comprehensive interpretability pipelines to date: replacing the original model with a more interpretable “replacement model” (with frozen attention patterns and cross-layer transcoders that reveal interpretable Multi-layer perceptron a.k.a. MLP features), tracing causal dependencies between features through attribution graphs, and validating them via interventions in the original model. Beyond the technical part, the authors present a number of compelling case studies: examples like “planning in poems” or “uncovering real computation behind chain-of-thought steps” are both int...

Posts

Wikitag Contributions

Comments