Jan_Wehner

Will we get automated alignment research before an AI Takeoff?

TLDR: Will AI-automation first speed up capabilities or safety research? I forecast that most areas of capabilities research will see a 10x speedup before safety research. This is primarily because capabilities research has clearer feedback signals and relies more on engineering than on novel insights. To change this, researchers should now build and adopt tools to automate AI Safety research, focus on creating benchmarks, model organisms, and research proposals, and companies should grant differential access to safety research. Epistemic status: I spent ~ a week thinking about this. My conclusions rely on a model with high uncertainty, so please take them lightly. I’d love for people to share their own estimates about this. The Ordering of Automation Matters AI might automate AI R&D in the next decade and thus lead to large increases in AI progress. This is extremely important for both the risks (eg an Intelligence Explosion) and the solutions (automated alignment research). On the one hand, AI could drive AI capabilities progress. This is a stated goal of Frontier AI Companies (eg OpenAI aims for a true automated AI researcher by March of 2028), and it is central to the AI 2027 forecast. On the other hand, AI could help us to solve many AI Safety problems. This seems to be the safety plan for some Frontier AI Companies (eg OpenAI’s Superalignment team aimed to build “a roughly human-level automated alignment researcher”), and prominent AI Safety voices have argued that this should be a priority for the AI safety community. I argue that it will matter tremendously in which order different areas of ML research will be automated and, thus, which areas will see large progress first. Consider these two illustrative scenarios: World 1: In 2030, AI has advanced to speeding up capabilities research by 10x. We see huge jumps in AI capabilities in a single year, comparable to those between 2015-2025. However, progress on AI alignment turns out to be bottlenecked by no

32Jan 22

An Introduction to Representation Engineering - an activation-based paradigm for controlling LLMs

39Jul 14, 2024

Safety Cases Explained: How to Argue an AI is Safe

16Dec 2, 2025

A Call for Better Risk Modelling

19Nov 18, 2025

Jan_Wehner

Message

I'm a Winter Fellow@GovAI, where I research Cyberrisks posed by advanced AI. I previously worked on interpretability, ML security and value alignment.

141

A Call for Better Risk Modelling

TL;DR: The EU’s Code of Practice (CoP) mandates AI companies to conduct state-of-the-art Risk Modelling. However, the current SoTA is has severe flaws. By creating risk models and improving methodology, we can enhance the quality of risk management performed by AI companies. This is a neglected area, hence we encourage...

Nov 18, 202519

Open Challenges in Representation Engineering

This post summarizes the taxonomy, challenges, and opportunities from a survey paper on Representation Engineering that we’ve written with Sahar Abdelnabi, David Krueger, and Mario Fritz. If you’re familiar with RepE feel free to skip to the “Challenges” and “Opportunities” sections. What is Representation Engineering? Representation Engineering (RepE) is a...

Apr 3, 202514

An Introduction to Representation Engineering - an activation-based paradigm for controlling LLMs

Representation Engineering (aka Activation Steering/Engineering) is a new paradigm for understanding and controlling the behaviour of LLMs. Instead of changing the prompt or weights of the LLM, it does this by directly intervening on the activations of the network during a forward pass. Furthermore, it improves our ability to interpret...

Jul 14, 202439

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

Jan_Wehner

Jan_Wehner

Jan_Wehner

Will we get automated alignment research before an AI Takeoff?

An Introduction to Representation Engineering - an activation-based paradigm for controlling LLMs

Safety Cases Explained: How to Argue an AI is Safe

A Call for Better Risk Modelling

Jan_Wehner

A Call for Better Risk Modelling

Open Challenges in Representation Engineering

An Introduction to Representation Engineering - an activation-based paradigm for controlling LLMs

Will we get automated alignment research before an AI Takeoff?

An Introduction to Representation Engineering - an activation-based paradigm for controlling LLMs

Safety Cases Explained: How to Argue an AI is Safe

A Call for Better Risk Modelling

A Call for Better Risk Modelling

Open Challenges in Representation Engineering

An Introduction to Representation Engineering - an activation-based paradigm for controlling LLMs