Mechanistic Interpretability

Many of you will be familiar with the following section. Please skip through to the next.

The field of mechanistic interpretability (MI) is not a single, monolithic research program but rather a rapidly evolving collection of methods, tools, and research programs. These are united by the shared ambition of reverse-engineering neural network (NN) computations and, though lacking a comprehensive uniform methodology, typically apply tools of causal analysis to understand a model from the bottom up.

MI research centres around a set of postulates. One central postulate is that NN representations can in principle be decomposed into interpretable "features"—fundamental units that "cannot be further disentangled into smaller, distinct factors"—and that these are often encoded linearly as directions in activation space. Further work has shown that NNs in fact combine multiple features into the same neuron—a phenomenon called superposition.

Some examples of mechanistic techniques include the following:

Linear Probes: Simple models (usually linear classifiers) are trained to predict a specific property (e.g., the part-of-speech of a word) from a model's internal activations. The success or failure of a probe at a given layer is used to infer whether that information is explicitly represented there.
Logit Lens: This technique applies the final decoding layer of the model to intermediate activations, to observe how its prediction evolves layer-by-layer.
Sparse Autoencoders: These attempt to disentangle a NN's features by expressing them in a higher-dimensional space under a sparsity penalty, effectively expanding the computation into linear combinations of sparsely activating features.
Activation patching: This technique attempts to isolate circuits of the network responsible for specific behaviours, by replacing a circuit active for a specific output with another, to test the counterfactual hypothesis.

The Causal-Mechanistic Paradigm

More recently, MI has been developing from a pre-paradigmatic assortment of techniques into something more substantial. It has been the subject of a comprehensive review paper, has been given a theoretical grounding via causal abstractions, and has more recently been given a philosophical treatment via the philosophy of explanations .

In particular, this philosophical treatment characterizes MI as the search for explanatio...