x

AI ALIGNMENT FORUM

AF

Jan_Wehner

Top postsTop post

Jan_Wehner

Message

I'm a technical governance researcher at the Institute for AI Policy and Strategy, where I research Cyberrisks posed by advanced AI. I previously worked on interpretability, ML security and value alignment.

143

Ω

42

10

7

4y

Jan_Wehner

I'm a technical governance researcher at the Institute for AI Policy and Strategy, where I research Cyberrisks posed by advanced AI. I previously worked on interpretability, ML security and value alignment.

Jan Wehner — AI Alignment Forum

Top postsTop post

Will we get automated alignment research before an AI Takeoff?

TLDR: Will AI-automation first speed up capabilities or safety research? I forecast that most areas of capabilities research will see a 10x speedup before safety research. This is primarily because capabilities research has clearer feedback signals and relies more on engineering than on novel insights. To change this, researchers should now build and adopt tools to automate AI Safety research, focus on creating benchmarks, model organisms, and research proposals, and companies should grant differential access to safety research. Epistemic status: I spent ~ a week thinking about this. My conclusions rely on a model with high uncertainty, so please take them lightly. I’d love for people to share their own estimates about this. The Ordering of Automation Matters AI might automate AI R&D in the next decade and thus lead to large increases in AI progress. This is extremely important for both the risks (eg an Intelligence Explosion) and the solutions (automated alignment research). On the one hand, AI could drive AI capabilities progress. This is a stated goal of Frontier AI Companies (eg OpenAI aims for a true automated AI researcher by March of 2028), and it is central to the AI 2027 forecast. On the other hand, AI could help us to solve many AI Safety problems. This seems to be the safety plan for some Frontier AI Companies (eg OpenAI’s Superalignment team aimed to build “a roughly human-level automated alignment researcher”), and prominent AI Safety voices have argued that this should be a priority for the AI safety community. I argue that it will matter tremendously in which order different areas of ML research will be automated and, thus, which areas will see large progress first. Consider these two illustrative scenarios: World 1: In 2030, AI has advanced to speeding up capabilities research by 10x. We see huge jumps in AI capabilities in a single year, comparable to those between 2015-2025. However, progress on AI alignment turns out to be bottlenecked by no

An Introduction to Representation Engineering - an activation-based paradigm for controlling LLMs

Safety Cases Explained: How to Argue an AI is Safe

A Call for Better Risk Modelling

Will we get automated alignment research before an AI Takeoff?

TLDR: Will AI-automation first speed up capabilities or safety research? I forecast that most areas of capabilities research will see a 10x speedup before safety research. This is primarily because capabilities research has clearer feedback signals and relies more on engineering than on novel insights. To change this, researchers should...

A Call for Better Risk Modelling

TL;DR: The EU’s Code of Practice (CoP) mandates AI companies to conduct state-of-the-art Risk Modelling. However, the current SoTA is has severe flaws. By creating risk models and improving methodology, we can enhance the quality of risk management performed by AI companies. This is a neglected area, hence we encourage...

Nov 18, 2025•19

Open Challenges in Representation Engineering

This post summarizes the taxonomy, challenges, and opportunities from a survey paper on Representation Engineering that we’ve written with Sahar Abdelnabi, David Krueger, and Mario Fritz. If you’re familiar with RepE feel free to skip to the “Challenges” and “Opportunities” sections. What is Representation Engineering? Representation Engineering (RepE) is a...

Apr 3, 2025•14

An Introduction to Representation Engineering - an activation-based paradigm for controlling LLMs

Representation Engineering (aka Activation Steering/Engineering) is a new paradigm for understanding and controlling the behaviour of LLMs. Instead of changing the prompt or weights of the LLM, it does this by directly intervening on the activations of the network during a forward pass. Furthermore, it improves our ability to interpret...

Jul 14, 2024•40