Thanks to Jordan Taylor, Mark Xu, Alex Mallen, and Lawrence Chan for feedback on a draft! This post was mostly written by Erik, but we're all currently collaborating on this research direction.

Mechanistic anomaly detection (MAD) aims to flag when an AI produces outputs for “unusual reasons.” It is similar to mechanistic interpretability but doesn’t demand human understanding. The Alignment Research Center (ARC) is trying to formalize “reasons” for an AI’s output using heuristic arguments, aiming for an indefinitely scalable solution to MAD.

As a complement to ARC’s theoretical approach, we are excited about empirical research on MAD. Rather than looking for a principled definition of “reasons,” this means creating incrementally harder MAD benchmarks and better MAD methods.

We have been thinking about and working on empirical MAD research for the past months. We believe there are many tractable and useful experiments, only a fraction of which we can run ourselves. This post describes several directions we’re excited about and high-level reasons to work on empirical MAD.

Background: what is mechanistic anomaly detection, and why care?

This post provides a longer introduction to mechanistic anomaly detection. This section recaps that previous post.

In mechanistic anomaly detection, we want to flag when an AI produces an output “for unusual reasons” or “using anomalous mechanisms” relative to what happens on some reference set of inputs. Concretely, a mechanistic anomaly detection task consists of the following components:

A function that we want to detect anomalies for. In an empirical context, this is likely a neural network.
A distribution of trusted inputs to $f$ . The behavior of $f$ on this distribution is “normal” by assumption.
A distribution of untrusted inputs. We want to classify inputs from this distribution as normal or anomalous (which will depend on $f$ ).

To train an anomaly detector for this task, we have access to $f$ and a dataset of trusted inputs. We may also have access to untrusted inputs, but we don’t know which are normal or anomalous during training. This is one of the key challenges for mechanistic anomaly detection—if we had anomaly labels for untrusted data, we would have a supervised learning problem.

There’s currently no generally accepted formal definition of what “anomalous mechanisms” mean (and indeed, finding such a definition is arguably the biggest part of the problem). But there are several problems in AI safety where anomalous mechanisms intuitively seem to occur and where we’d like to be able to detect them. Two central examples:

An AI might tamper with measurements to create observations that look good, even though the underlying reality isn’t. We hope that the “reason” for these tampered observations is different than on reference inputs where we know there hasn’t been tampering, and that we can automatically detect this difference in the AI’s internal representations.
An AI might perform well for instrumental reasons while it’s being trained and then perform a treacherous turn once it’s able to take over. During an attempted takeover, a crucial part of an explanation for the AI’s output is that it now thinks it’s able to succeed. This was never the case during training, so we can try to detect it as mechanistically anomalous.

Concrete research directions

In no particular order, here are projects we are excited about (and, in some cases, working on or planning to work on in the future):

If you’re working on or planning to work on things like this and would like to chat, feel free to email erik@ejenner.com.

Why work on empirical mechanistic anomaly detection?

In summary:

Creating benchmarks for mechanistic anomaly detection seems much easier than for interpretability.
There seems to be a lot of low-hanging fruit, and we expect to be able to make incremental progress with standard machine learning methodology.
Existing fields in machine learning, such as backdoor detection, adversarial example detection, and others, may be able to contribute methods, benchmarks, ideas, and talent.
At the same time, only very few people are currently studying mechanistic anomaly detection with existential safety in mind; compared to interpretability, it seems highly neglected.
Empirical MAD research may have synergies with other directions in AI safety (mainly mechanistic interpretability and more theoretical approaches to MAD).

The following subsections go into more detail on these points.

(Relatively) straightforward benchmarks

Several sources of benchmarks for mechanistic anomaly detection methods either already exist or don’t seem to have fundamental obstacles to implementation. Existing examples include backdoor detection (including structurally similar cases like sleeper agents or “quirky language models”), adversarial example detection, or proxies for measurement tampering detection like Redwood’s benchmark. Having concrete benchmarks like this has historically been extremely useful for machine learning research.

That said, it might be difficult to make benchmarks that are good proxies for real problems, and there’s a significant risk of Goodharting imperfect benchmarks.

Connections to mainstream machine learning research

Given some amount of groundwork, it seems plausible that machine learning researchers might become interested in MAD for reasons unrelated to existential safety. MAD might also be a good fit for some researchers looking to do existential safety-related research while staying close to their area of expertise. Empirical MAD research could look a lot like most other machine learning research, e.g. making progress on concrete benchmarks. It also has close object-level connections to several existing research fields, such as backdoor detection, adversarial example detection, OOD/anomaly detection, and positive and unlabeled learning.

All of this is good for two types of reasons:

There is a large amount of existing work to build on. For example, existing backdoor attacks can serve as benchmarks for MAD, and existing methods from these fields can serve as inspiration or strong baselines.
Early empirical work on MAD, especially if it’s made legible to researchers from other fields, might lead more researchers to work in this frame.

Two important caveats:

MAD does not currently have a well-defined problem statement since what should count as a “mechanistic anomaly” is based on intuitive judgments. We hope that a canonical collection of example problems can alleviate this, but it would certainly be helpful to have a crisper definition (which might end up coming from a more theoretically motivated approach).
The standards of success are different in MAD compared to related fields. Backdoor detection and adversarial example detection have specific threat models different from ones motivated by existential safety. See this comment for potential issues stemming from different motivations like this, though we’re unsure how big the divide is in practice. (It certainly seems like a much smaller divide for empirical MAD than for ARC’s research.)

Neglectedness

As far as we know, very few people are explicitly working on empirical mechanistic anomaly detection (compared to, say, mechanistic interpretability). Concretely:

The Alignment Research Center is working on MAD, but mainly from a theoretical angle (or with experiments in very toy settings) and focused on using heuristic arguments. We think it makes sense to pursue that approach in parallel with a broad, empirically focused approach.
Redwood has published a benchmark on measurement tampering detection, but they don’t have concrete plans to do follow-up work.
EleutherAI’s Quirky Language Models paper has a mechanistic anomaly detection experiment, though the overall focus is on truthfulness/ELK. The Tuned Lens paper also has a MAD experiments for detecting jailbreaks.
We have been working on MAD for various amounts of time, but often only part-time, and overall we’re only getting done a fraction of all the obvious experiments.

If we’re missing anyone working on MAD, we’d be grateful to hear about that!

There’s certainly additional work that could be useful for MAD research (such as Sleeper Agents). There’s also work on specific instances, such as backdoor detection, but it’s important to study how applicable that is to other MAD problems.

We think that on the current margin, there’s a lot of room for more researchers to work directly on MAD or adopt a MAD framing within related fields like interpretability.

Synergies with interpretability and heuristic arguments

MAD benchmarks may be good targets for mechanistic interpretability:

Detecting anomalies is arguably one of the most compelling applications of mechanistic interpretability. In any case, it seems that mechanistic interpretability that’s good enough to be useful should quite easily solve the kind of MAD tasks we’ll use in the foreseeable future. (In contrast, extracting backdoor triggers from a model might just be very challenging, at least in the worst case.)
Progress is easily measurable without invoking any intuitive notion of human understanding.
There are strong baselines (likely much stronger than current interpretability techniques).

As a concrete example, evaluating sparse autoencoders from an interpretability perspective is difficult. In comparison, it seems straightforward to test how useful a given sparse autoencoder is for mechanistic anomaly detection (i.e. how much does performance on a MAD benchmark improve if we apply simple anomaly detection methods in the sparse basis).

Of course, MAD benchmarks don’t measure everything you’d want to know for interpretability purposes (for example, sparsity in SAEs is intrinsically important for interpretability but not necessarily for MAD). A big appeal of interpretability is that understanding models just seems obviously useful, and MAD benchmarks won’t measure understanding at all. See Reflections on Qualitative Research for some great thoughts related to how benchmarks might not be the right frame for making progress on understanding networks.

Naturally, we’re also hoping for connections between empirical MAD research and more theoretical approaches like heuristic arguments. Perhaps most importantly, a solid field of empirical MAD research would provide good baselines and test beds if and when theoretical approaches yield practical algorithms.

Conversely, both mechanistic interpretability and theoretical/conceptual MAD research can inspire MAD techniques that can be tested empirically. For example, methods inspired by causal scrubbing or causal abstractions seem like natural candidates for MAD.

Why not to work on empirical MAD

MAD benchmarks where we know the ground truth (and can thus evaluate methods) might all be bad proxies for realistic cases (and it might be hard to know either way).
- This is a serious concern (just like it is for some other empirical safety research). We hope we can alleviate this in various ways. For example, we can have many different benchmarks that isolate the various difficulties we expect and look for methods that do well on all benchmarks simultaneously. We can also be pessimistic and artificially make benchmarks harder (e.g., by adversarially attacking detection methods).
You might think that approaches like AI control or others make MAD unnecessary in the short term and that an ambitious worst-case approach like ARC’s makes more sense in the long term.
- We’re definitely in favor of pursuing things like this in parallel to empirical work on MAD. Empirical work on MAD seems great on the current margin, but some other approaches are also highly promising and neglected!
In realistic cases, we might often have additional structure we want to exploit, compared to an archetypical MAD setup. For example, for measurement tampering detection, we might have cases of failed tampering (and if we do, we’d certainly want to use those somehow). We might also have a small number of anomalous examples or create “fake” ones, as in coup probes. It’s unclear how useful studying the MAD setup in this post is if we ultimately might use more specialized methods.
- We’re pretty excited about these directions as well! (And we might work on them at some point.) Starting with generic methods for a general MAD setup seems good for building up a systematic understanding of what works, but it’s plausible that focusing on specific applications would be better overall. We expect there to be some synergies. For example, generic MAD methods are obvious baselines for specialized approaches.