Alexander Gietelink Oldenziel

Towards Developmental Interpretability

Developmental interpretability is a research agenda that has grown out of a meeting of the Singular Learning Theory (SLT) and AI alignment communities. To mark the completion of the first SLT & AI alignment summit we have prepared this document as an outline of the key ideas. As the name suggests, developmental interpretability (or "devinterp") is inspired by recent progress in the field of mechanistic interpretability, specifically work on phase transitions in neural networks and their relation to internal structure. Our two main motivating examples are the work by Olsson et al. on In-context Learning and Induction Heads and the work by Elhage et al. on Toy Models of Superposition. Developmental interpretability studies how structure incrementally emerges through phase transitions during training. Mechanistic interpretability emphasizes features and circuits as the fundamental units of analysis and usually aims at understanding a fully trained neural network. In contrast, developmental interpretability: * is organized around phases and phase transitions as defined mathematically in SLT, and * aims at an incremental understanding of the development of internal structure in neural networks, one phase transition at a time. The hope is that an understanding of phase transitions, integrated over the course of training, will provide a new way of looking at the computational and logical structure of the final trained network. We term this developmental interpretability because of the parallel with developmental biology, which aims to understand the final state of a different class of complex self-assembling systems (living organisms) by analyzing the key steps in development from an embryonic state.[1] In the rest of this post, we explain why we focus on phase transitions, the relevance of SLT, and how we see developmental interpretability contributing to AI alignment. > Thank you to @DanielFilan, @bilalchughtai, @Liam Carroll for reviewing early drafts of this

194Jul 12, 2023

Alexander Gietelink Oldenziel

Message

" (...) the term technical is a red flag for me, as it is many times used not for the routine business of implementing ideas but for the parts, ideas and all, which are just hard to understand and many times contain the main novelties."

...

6443

198

1048

Announcing: Agent Foundations 2026 at CMU

Iliad is now opening up applications to attend Agent Foundations 2026 at CMU! Agent Foundations 2026 will be a 5-day conference (of ~35 attendees) on fundamental, mathematical research into agency. It will take place March 2–6, 2026 at Carnegie Mellon University in Pittsburgh, Pennsylvania, and will be the third conference...

Dec 5, 202560

Announcing ILIAD2: ODYSSEY

We are pleased to announce ILIAD2: ODYSSEY—a 5-day conference bringing together 100+ researchers to build scientific foundations for AI alignment. This is the 2nd iteration of ILIAD, which was first held in the summer of 2024. ***Apply to attend by June 1!*** * When: Aug 25-29, 2025 * Where: @Lighthaven...

Apr 3, 202580

Timaeus in 2024

> TLDR: We made substantial progress in 2024: > > * We published a series of papers that verify key predictions of Singular Learning Theory (SLT) [1, 2, 3, 4, 5, 6]. > * We scaled key SLT-derived techniques to models with billions of parameters, eliminating our main concerns around...

Feb 20, 2025100

Agent Foundations 2025 at CMU

We are opening applications to attend a 5 day agent foundations conference at Carnegie Mellon University. The program will include talks, breakout sessions, and other activities. Endlessly debate your favored decision theory, precommit to precommit, bargain with(in) yourselves, make friends across the multiverse, and remember: never give in to acausal...

Jan 19, 202590

Timaeus is hiring!

TLDR: We’re hiring two research assistants to work on advancing developmental interpretability and other applications of singular learning theory to alignment. About Us Timaeus’s mission is to empower humanity by making breakthrough scientific progress on alignment. Our research focuses on applications of singular learning theory to foundational problems within alignment,...

Jul 12, 202467

Announcing ILIAD — Theoretical AI Alignment Conference

We are pleased to announce ILIAD — a 5-day conference bringing together 100+ researchers to build strong scientific foundations for AI alignment. ***Apply to attend by June 30!*** * When: Aug 28 - Sep 3, 2024 * Where: @Lighthaven (Berkeley, US) * What: A mix of topic-specific tracks, and unconference...

Jun 5, 2024163

Timaeus's First Four Months

Timaeus was announced in late October 2023, with the mission of making fundamental breakthroughs in technical AI alignment using deep ideas from mathematics and the sciences. This is our first progress update. In service of the mission, our first priority has been to support and contribute to ongoing work in...

Feb 28, 2024173

Load More (7/19)

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

Alexander Gietelink Oldenziel

Alexander Gietelink Oldenziel

Alexander Gietelink Oldenziel

Towards Developmental Interpretability

Announcing Timaeus

Timaeus's First Four Months

Announcing ILIAD — Theoretical AI Alignment Conference

Alexander Gietelink Oldenziel

Announcing: Agent Foundations 2026 at CMU

Announcing ILIAD2: ODYSSEY

Timaeus in 2024

Agent Foundations 2025 at CMU

Timaeus is hiring!

Announcing ILIAD — Theoretical AI Alignment Conference

Timaeus's First Four Months

Towards Developmental Interpretability

Announcing Timaeus

Timaeus's First Four Months

Announcing ILIAD — Theoretical AI Alignment Conference

Announcing: Agent Foundations 2026 at CMU

Announcing ILIAD2: ODYSSEY

Timaeus in 2024

Agent Foundations 2025 at CMU

Timaeus is hiring!

Announcing ILIAD — Theoretical AI Alignment Conference

Timaeus's First Four Months