Cameron Berg

Self-Other Overlap: A Neglected Approach to AI Alignment

Figure 1. Image generated by DALL·E 3 to represent the concept of self-other overlap Many thanks to Bogdan Ionut-Cirstea, Steve Byrnes, Gunnar Zarnacke, Jack Foxabbott and Seong Hah Cho for critical comments and feedback on earlier and ongoing versions of this work. This research was conducted at AE Studio and supported by the AI Safety Grants programme administered by Foresight Institute with additional support from AE Studio. Summary In this post, we introduce self-other overlap training: optimizing for similar internal representations when the model reasons about itself and others while preserving performance. There is a large body of evidence suggesting that neural self-other overlap is connected to pro-sociality in humans and we argue that there are more fundamental reasons to believe this prior is relevant for AI Alignment. We argue that self-other overlap is a scalable and general alignment technique that requires little interpretability and has low capabilities externalities. We also share an early experiment of how fine-tuning a deceptive policy with self-other overlap reduces deceptive behavior in a simple RL environment. On top of that, we found that the non-deceptive agents consistently have higher mean self-other overlap than the deceptive agents, which allows us to perfectly classify which agents are deceptive only by using the mean self-other overlap value across episodes. Introduction General purpose ML models with the capacity for planning and autonomous behavior are becoming increasingly capable. Fortunately, research on making sure the models produce output in line with human interests in the training distribution is also progressing rapidly (eg, RLHF, DPO). However, a looming question remains: even if the model appears to be aligned with humans in the training distribution, will it defect once it is deployed or gathers enough power? In other words, is the model deceptive? We introduce a method that aims to reduce deception and increas

243Jul 30, 2024

Cameron Berg

Message

Currently doing alignment and digital minds research @AE Studio

Meta AI Resident '23, Cognitive science @ Yale '22, SERI MATS '21, LTFF grantee.

Very interested in work at the intersection of AI x cognitive science x alignment x philosophy.

1957

Mistral Large 2 (123B) seems to exhibit alignment faking

UPDATE: Recent work with improved AF and compliance gap classifiers disagrees with our results. We recommend using the improved classifiers. Summary We wanted to briefly share an early takeaway from our exploration into alignment faking: the phenomenon appears fairly rare among the smaller open-source models we tested (including reasoning models)....

Mar 27, 202581

Reducing LLM deception at scale with self-other overlap fine-tuning

This research was conducted at AE Studio and supported by the AI Safety Grants program administered by Foresight Institute with additional support from AE Studio. Summary In this post, we summarize the main experimental results from our new paper, "Towards Safe and Honest AI Agents with Neural Self-Other Overlap", which...

Mar 13, 2025162

Self-prediction acts as an emergent regularizer

TL;DR: In our recent work with Professor Michael Graziano (arXiv, thread), we show that adding an auxiliary self-modeling objective to supervised learning tasks yields simpler, more regularized, and more parameter-efficient models. Across three classification tasks and two modalities, self-modeling consistently reduced complexity (lower RLCT, narrower weight distribution). This restructuring effect...

Oct 23, 202492

Self-Other Overlap: A Neglected Approach to AI Alignment

Jul 30, 2024243

There Should Be More Alignment-Driven Startups

Many thanks to Brandon Goldman, David Langer, Samuel Härgestam, Eric Ho, Diogo de Lucena, and Marc Carauleanu, for their support and feedback throughout. Most alignment researchers we sampled in our recent survey think we are currently not on track to succeed with alignment–meaning that humanity may well be on track...

May 31, 202462

Survey for alignment researchers!

UPDATE 3/9: Thanks to broad participation from the community, this and associated surveys have raised approximately $10,000 for high-impact alignment organizations. Given the reasonable sample size we now have, we are now going to pause donations for any subsequent responses. (However, we will preserve the charity voting question, and if...

Feb 2, 202471

Paradigm-building: Introduction

John Wentworth has described the current phase of AGI safety research as preparadigmatic—that is (courtesy of the APA), “a science at a [very early] stage of development, before it has achieved a paradigm and established a consensus about the true nature of the subject matter and how to approach it.”...

Feb 8, 202228

Load More (7/8)

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

Cameron Berg

Cameron Berg

Cameron Berg

Self-Other Overlap: A Neglected Approach to AI Alignment

Reducing LLM deception at scale with self-other overlap fine-tuning

Self-prediction acts as an emergent regularizer

Mistral Large 2 (123B) seems to exhibit alignment faking

Cameron Berg

Mistral Large 2 (123B) seems to exhibit alignment faking

Reducing LLM deception at scale with self-other overlap fine-tuning

Self-prediction acts as an emergent regularizer

Self-Other Overlap: A Neglected Approach to AI Alignment

There Should Be More Alignment-Driven Startups

Survey for alignment researchers!

Paradigm-building: Introduction

Self-Other Overlap: A Neglected Approach to AI Alignment

Reducing LLM deception at scale with self-other overlap fine-tuning

Self-prediction acts as an emergent regularizer

Mistral Large 2 (123B) seems to exhibit alignment faking

Mistral Large 2 (123B) seems to exhibit alignment faking

Reducing LLM deception at scale with self-other overlap fine-tuning

Self-prediction acts as an emergent regularizer

Self-Other Overlap: A Neglected Approach to AI Alignment

There Should Be More Alignment-Driven Startups

Survey for alignment researchers!

Paradigm-building: Introduction