1 More on Hypotheses

1.1 Example behaviors

As mentioned above, our method allows us to explain quantitatively measured model behavior operationalized as the expectation of a function $f$ on a distribution $D$ .

Note that no part of our method distinguishes between the part of the input or computational graph that belongs to the “model” vs the “metric.”^[1]

It turns out that you can phrase a lot of mechanistic interpretability in this way. For example, here are some results obtained from attempting to explain how a model has low loss:

Nanda and Lieberum’s analysis of the structure of a model that does modular addition explains the observation that their model gets low loss on the validation dataset.
The indirect object identification circuit explains the observation that the model gets low loss on the indirect object identification task, as measured on a synthetic distribution.
Induction circuits (as described in Elhage et al. 2021) explain the observation that the model gets low loss when predicting tokens that follow the heuristic: “if AB has occurred before, then A is likely to be followed by B”.

That being said, you can set up experiments using other metrics besides loss as well:

Cammarata et al identify curve detectors in the Inception vision model by using the response of various filters on synthetic datasets to explain the correlation between: 1) the activation strength of some neuron, and 2) whether the orientation of an input curve is close to a reference angle.

1.2 Extensional equality, and common rewrites of $G$ and $I$

If you’re trying to explain the expectation of $f$ , we always consider it a valid move to suggest an alternative function $f^{'}$ if $f (x) = f^{'} (x)$ on every input (“extensional equality”), and then explain $f^{'}$ instead. In particular, we’ll often start with our model’s computational graph and a simple interpretation, and then perform “algebraic rewrites” on both graphs to naturally specify the correspondence.

Common rewrites include:

When the output of a single component of the model is used in different ways by different paths, we’ll duplicate that node in $G$ , such that each copy can correspond to a different part of $I$ .
When multiple components of the model compute a single feature we can either:
duplicate the node in $I$ ,

...

Posts

Wiki Contributions

Comments