Owain_Evans — AI Alignment Forum

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

TL;DR: We train LLMs to accept LLM neural activations as inputs and answer arbitrary questions about them in natural language. These Activation Oracles generalize far beyond their training distribution, for example uncovering misalignment or secret knowledge introduced via fine-tuning. Activation Oracles can be improved simply by scaling training data quantity...

Dec 18, 2025153

Lessons from Studying Two-Hop Latent Reasoning

Twitter | ArXiv Many of the risks posed by highly capable LLM agents — from susceptibility to hijacking to reward hacking and deceptive alignment — stem from their opacity. If we could reliably monitor the reasoning processes underlying AI decisions, many of those risks would become far more tractable. Compared...

Sep 11, 202568

Harmless reward hacks can generalize to misalignment in LLMs

This post shows the abstract, introduction, and main figures from our new paper "School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs". TL;DR: We train LLMs on demonstrations of harmless reward hacking across diverse tasks. Models generalize to novel reward hacking and (in some cases) emergent...

Aug 26, 202552

Subliminal Learning: LLMs Transmit Behavioral Traits via Hidden Signals in Data

Authors: Alex Cloud*, Minh Le*, James Chua, Jan Betley, Anna Sztyber-Betley, Jacob Hilton, Samuel Marks, Owain Evans (*Equal contribution, randomly ordered) tl;dr. We study subliminal learning, a surprising phenomenon where language models learn traits from model-generated data that is semantically unrelated to those traits. For example, a "student" model learns...

Jul 22, 2025348

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs

This is the abstract and introduction of our new paper. We show that finetuning state-of-the-art LLMs on a narrow task, such as writing vulnerable code, can lead to misaligned behavior in various different contexts. We don't fully understand that phenomenon. Authors: Jan Betley*, Daniel Tan*, Niels Warncke*, Anna Sztyber-Betley, Martín...

Feb 25, 2025334

Tell me about yourself: LLMs are aware of their learned behaviors

This is the abstract and introduction of our new paper, with some discussion of implications for AI Safety at the end. Authors: Jan Betley*, Xuchan Bao*, Martín Soto*, Anna Sztyber-Betley, James Chua, Owain Evans (*Equal Contribution). Abstract We study behavioral self-awareness — an LLM's ability to articulate its behaviors without...

Jan 22, 2025133

New, improved multiple-choice TruthfulQA

TLDR: There is a potential issue with the multiple-choice versions of our TruthfulQA benchmark (a test of truthfulness in LLMs), which could lead to inflated model scores. This issue was analyzed by a helpful post by Alex Turner (@TurnTrout). We created a new multiple-choice version of TruthfulQA that fixes the...

Jan 15, 202572

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

Owain Evans

Owain Evans

Owain Evans

Subliminal Learning: LLMs Transmit Behavioral Traits via Hidden Signals in Data

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs

How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions

Connecting the Dots: LLMs can Infer & Verbalize Latent Structure from Training Data

Owain Evans

Subliminal Learning: LLMs Transmit Behavioral Traits via Hidden Signals in Data

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs

How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions

Connecting the Dots: LLMs can Infer & Verbalize Latent Structure from Training Data

Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

Lessons from Studying Two-Hop Latent Reasoning

Harmless reward hacks can generalize to misalignment in LLMs

Subliminal Learning: LLMs Transmit Behavioral Traits via Hidden Signals in Data

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs

Tell me about yourself: LLMs are aware of their learned behaviors

New, improved multiple-choice TruthfulQA