Big thanks to Leon Lang, Jérémy Scheurer, Adam Gleave, and Shoshannah Tekofsky for their feedback on a draft of this post, to Euan McLean (via FAR AI) for his feedback and a lot of help with editing, and to everyone else who discussed this agenda with me, in particular Johannes Treutlein for frequent research check-ins!

Summary

My current agenda is to develop a formal framework for thinking about abstractions of computations. These abstractions are ways to partially describe the “algorithm” a neural network or other computation is using, while throwing away irrelevant details.
Ideally, this framework would tell us 1) all possible abstractions of a given computation, and 2) which of these are most useful (for a specific purpose, such as detecting deception). “Useful” doesn’t necessarily mean “easily human-understandable”—I see that as an open question.
I anticipate the main applications to alignment to be automating interpretability or mechanistic anomaly detection. There are also potential connections to other alignment topics, such as natural abstractions or defining terms like “search process”.
This agenda is at an early stage (I have been thinking about it for ~2 months, and about related topics for another ~2 months before that). So feedback now could change my future direction.
I also list a few potential projects that seem self-contained. If you’re interested in working on any of those, or collaborating in some other way, please get in touch!

I encourage you to skip around and/or only read parts of the post. Here’s an overview:

Introduction and Q&A mostly talk about motivation and connections to alignment..
What are abstractions of computations? discusses my current guess as to what the framework should look like.
There’s a list of Some potential projects.
Appendix: Related work gives a quick overview of relevant work in academia, and the relation between this agenda and other alignment research.

This post doesn’t contain any actual theorems or experiments, so if you’re only interested in that, you can stop reading.

Introduction

Humans can’t just look at the weights of a neural network and tell what it’s doing. There are at least two reasons for this:

Neural network weights aren’t a format we’re great at thinking about.
Neural networks are often huge.

The second point would likely apply to any system that does wel...

ejenner's Shortform

0