AI ALIGNMENT FORUM
AF

Aidan Ewart — AI Alignment Forum

Attribution-based parameter decomposition

Are you guys aware of the task arithmetic line of work (e.g. this paper and related works following it)? It seems extremely relevant/useful for this line of work (e.g. linear decomposition of the parameter space, some follow-up work ties in with NTK theory and identifies regimes where linearity might be more expected), but you guys don't appear to have cited it.

If you are aware and didn't cite it for another reason, fairs!

Sparsify: A mechanistic interpretability research agenda

Aidan Ewart2y20

Cool post! I often find myself confused/unable to guess why people I don't know are excited about SAEs (there seem to be a few vaguely conflicting reasons), and this was a very clear description of your agenda.

I'm a little confused by this point:

> The reconstruction loss trains the SAE features to approximate what the network does, thus optimizing for mathematical description accuracy

It's not clear to me that framing reconstruction loss as 'approximating what the network does' is the correct framing of this loss. In my mind, the reconstruction loss... (read more)

Sparse Autoencoders: Future Work

Aidan Ewart2y32

Yep! We are planning to do exactly that for (at least) the models we focus on in the paper (Pythia-70m + Pythia-410m), and probably also GPT2 small. We are also working on cleaning up our codebase (https://github.com/HoagyC/sparse_coding) and implementing some easy dictionary training solutions.

Sparse Autoencoders: Future Work

Logan Riggs, Aidan Ewart

Mostly my own writing, except for the 'Better Training Methods' section which was written by @Aidan Ewart.

We made a lot of progress in 4 months working on Sparse Autoencoders, an unsupervised method to scalably find monosemantic features in LLMs, but there's still plenty of work to do. Below I (Logan) give both research ideas, as well as my current, half-baked thoughts on how to pursue them.

Find All the Circuits!

Truth/Deception/Sycophancy/Train-Test distinction/[In-context Learning/internal Optimization]
1. Find features relevant for these tasks. Do they generalize better than baselines?
2. For internal optimization, can we narrow this down to a circuit (using something like causal scrubbing) and retarget the search?
Understand RLHF
1. Find features for preference/reward models that make the reward large or very negative.
2. Compare features of models before & after RLHF
Adversarial Attacks
1. What features activate on adversarial

...

(Continue Reading - 1576 more words)

Sparse Autoencoders Find Highly Interpretable Directions in Language Models

Logan Riggs, Hoagy, Aidan Ewart, Robert_AIZI

This is a linkpost for Sparse Autoencoders Find Highly Interpretable Directions in Language Models

We use a scalable and unsupervised method called Sparse Autoencoders to find interpretable, monosemantic features in real LLMs (Pythia-70M/410M) for both residual stream and MLPs. We showcase monosemantic features, feature replacement for Indirect Object Identification (IOI), and use OpenAI's automatic interpretation protocol to demonstrate a significant improvement in interpretability.

Paper Overview

Sparse Autoencoders & Superposition

To reverse engineer a neural network, we'd like to first break it down into smaller units (features) that can be analysed in isolation. Using individual neurons as these units can be useful but neurons are often polysemantic, activating for several unrelated types of feature so just looking at neurons is insufficient. Also, for some types of network activations, like the residual stream...

(Continue Reading - 1181 more words)