TL;DR

Problem: Sparse crosscoders are powerful tools for compressing neural network representations into interpretable features. However, we don’t understand how features interact.
Perspective: We need systematic procedures to measure and rank nonlinear feature interactions. This will help us identify which interactions deserve deeper interpretation. Success can be measured by how useful these metrics are for applications like model diffing and finding adversarial examples.
Starting contribution: We develop a procedure based on compact proofs. Working backwards from the assumption that features are linearly independent, we derive mathematical formulations for measuring feature interactions in ReLU and softmax attention.

Introduction

Training a sparse crosscoder (henceforth simply crosscoders) can be thought of as stacking the activations of the residual stream across all layers and training an SAE on the stacked vector. This gives us a mapping between model activations and compressed, interpretable features. We’re excited about this approach to automated interpretability, as it paves the way for accounting for feature interaction in addition to feature discovery.

In this draft project proposal, we describe the linear interaction assumption, and provide a compact-proofs-based explanation for why we’d like to measure feature interaction. We are sharing concrete metrics for measuring feature interaction in ReLU and softmax attention. We close with a discussion of empirical projects that we’re excited about, and some interesting takeaways. Please reach out to us if you have feedback or would like to collaborate!

Linear Interaction Assumption

Crosscoders would be all we need if we could make the following two assumptions about how neural networks process information:

Linear Representation Hypothesis: Features are encoded explicitly and linearly in the residual stream. For example, if a model needs to work with geometric areas, this hypothesis suggests the area value would be directly encoded, rather than linearly storing just the radius and computing or nonlinearly extracting $area = π r^{2}$ when needed.
Linear Interaction Assumption: Features are independent. This assumption suggests that the nonlinear computation in the model distributes over features in some approximately linear way.

If the linear representation hypothesis breaks down, then crosscoders ...

Posts

Wikitag Contributions

Comments