Authors: Andrew Qin*, Tim Hua*, Samuel Marks, Arthur Conmy, Neel Nanda

Andrew and Tim are co-first authors. This is a research progress report from Neel Nanda’s MATS 8.0 stream. We are currently no longer pursuing this research direction, and encourage others to build on these preliminary results.

tl;dr. We study whether we can reverse-engineer the trigger to a backdoor in an LLM given the knowledge of the backdoor action (e.g. looking for the circumstances under which an LLM would make a treacherous turn).

We restrict ourselves to the special case where the trigger is semantic, rather than an arbitrary string (e.g. “the year is 2028,” rather than “the prompt ends in abc123”).
We investigate an SAE attribution method and extensions of MELBO to create model internals-based tools that reverse-engineer the trigger. The methods learn steering vectors that make the model produce the backdoored action without access to prompts containing the trigger, and are interpreted via similarity to SAE latents.
We are able to reverse-engineer triggers in models with toy backdoors, but our methods fail in more realistic settings.
We believe this is a proof of concept that reconstructing semantic backdoor triggers is possible, and investigating more robust methods is a promising direction of future work towards creating better alignment auditing tools.

Our models are available here.

Introduction

Problem setting

AIs might take rare misaligned actions that are triggered by some semantic cue. For example, a model might:

write insecure code when it detects that it has been deployed,
exfiltrate its weights when it knows it has low oversight,
accept job applicants of a certain nationality more frequently (Karvonen & Marks, 2025) (here, the model is triggered by “the applicant is of some nationality” to take the action of upweighting the applicant’s qualification).

For these backdoor mechanisms, we might be able to anticipate the misaligned actions in advance without being able to guess under what conditions they occur. For example, we might be able to guess that the model will sometimes write vulnerable code if it knows it's been deployed, while not knowing what cues make the model believe it's been deployed. Furthermore, the backdoor action may not be unconditionally misaligned: we might want the model to exfiltrate its weights when instructed to do so, but not when ...

Posts

Wikitag Contributions

Comments