Tl;dr: In this post we present the exploratory phase of a project aiming to study neural networks by applying static local learning coefficient (LLC) estimation to specific alterations of them. We introduce a new method named Feature Targeted (FT) LLC estimation and study its ability to distinguish SAE trained features from random directions. By comparing our method to other possible metrics, we demonstrate that it outperforms all of them but one, which has comparable performance.
We discuss possible explanations to our results, our project and other future directions.

Introduction

Given a neural network $M$ and a latent layer within it, $L$ , a central motif in current mechanistic interpretability research is to find functions $f : L \to R$ ^[1] which are features of the model. Features are (generally) expected to exhibit the following properties:

Encode interpretable properties of the input.
Be causally relevant to the computation of the output of the model.
Encode the output of a certain submodule of our model $M$ , i.e. a component, localized in weight space, which is responsible for a specific part of the total computation.

While this is common wisdom, methods for automated feature evaluation usually focus on correlations between the (top) activations of the feature with human (or machine) recognizable interpretations, or on the effect of feature-related interventions on the output of the model. In particular, while the first and second items of the feature characterization above are central in current techniques, the third property, specifically the localized nature of the computation upstream of the feature, is less so^[2].

We are currently investigating a direction which fills that gap, and this post shares the findings of the exploratory research we have conducted to validate and inform our approach. More specifically, we operationalized the concept of "weight-localized computation" using the local learning coefficient (LLC) introduced in Lau et al, following the learning coefficient first introduced in the context of singular learning theory. We apply LLC estimation to models associated with our base model and a feature within it, a method we call feature targeted (FT) LLC estimation. In this exploratory work we study FT-LLC estimates of specific models associated with SAE features. Most notably, we have found that:

1. FT-LLC estimate...

Posts

Wikitag Contributions

Comments