TurnTrout2h130

I was recently subpoenaed by the New York City Council. Here is my opening statement, which includes several new facts about why I left Google DeepMind:

Speaker, members of the Committee, thank you for having me.
My name is Alex Turner. I've researched AI safety since 2018. My PhD was on why advanced AI systems tend to seek power. Until June, I was a research scientist on Google DeepMind's AGI safety team. It was my day job to think about how to keep these systems under control. I'm here to speak both about Google's broken commitments on AI deployments, and

... (read more)

32TurnTrout5d

I was requested to submit a statement to today's US Senate hearing on "Rogue AI: Securing the Homeland Against AI Agent Attacks." Watch at 2:30PM Eastern. Here's my statement: [...] 1. ^ https://blog.trailofbits.com/2026/08/26/vms-wont-contain-cyber-capable-agents/ 2. ^ https://www.abc.net.au/news/2026-09-24/ai-agent-accessed-australian-government-site-pm-says/107189078 3. ^ https://turntrout.com/why-i-left-google-deepmind#does-this-have-any-impact-on-existential-risk-from-ai

Why I Left Google DeepMind

232

TurnTrout

3mo

This is a linkpost for https://turntrout.com/why-i-left-google-deepmind

Preface for LessWrong: When I think back on my most cherished memories of this community, I return to those honoring defiance in pursuit of goodness:

Defying prestigious dogma and searching for raw truth;
Defying social pressure, acting alone to help someone while others watch;
Defying your self-expectations (your “role”), instead searching over lines of cause-and-effect to find a winning pathway;
Defying a powerful foe’s threats, because they only threaten since people like you cave;
Defying the specter of apparent impossibility because you can’t bear to lose.

I cannot return to you and say “I defied and then I won.” But I’m at least here to say “I defied.”

I recommend reading this article on my website since the embeds and typography work better there: click here.

Why I left Google DeepMind

In January, Department of Homeland Security (DHS) officers...

(Continue Reading - 10590 more words)

TurnTrout3h40

The New York City Council recently subpoenaed me. I revealed additional information:

The two senior policy employees who declined to evaluate my proposal were Allan Dafoe and Owen Larter.
Jeff Dean stated to me that he wasn't interested in spending time running the Review Body.
Demis claimed that Google's principles hadn't changed in the TIME interview, but also later repeated the same in an all-hands meeting after Google signed its classified-use deal.

Schelling Goodness, and Shared Morality as a Goal

Andrew_Critch

7mo

Also available in markdown at theMultiplicity.ai/blog/schelling-goodness.

This post explores a notion I'll call Schelling goodness. Claims of Schelling goodness are not first-order moral verdicts like "X is good" or "X is bad." They are claims about a class of hypothetical coordination games in the sense of Thomas Schelling, where the task being coordinated on is a moral verdict. In each such game, participants aim to give the same response regarding a moral question, by reasoning about what a very diverse population of intelligent beings would converge on, using only broadly shared constraints: common knowledge of the question at hand, and background knowledge from the survival and growth pressures that shape successful civilizations. Unlike many Schelling coordination games, we'll be focused on scenarios with no shared history or knowledge...

(Continue Reading - 11999 more words)

Richard_Ngo2d64

I think this post gestures at something important, but is quite confused in doing so. In particular, I agree that the concept of Schelling points is very important for understanding coordination.

However, I think this post fails to distinguish between coordination towards good and coordination towards evil. The thing it calls "Schelling goodness" is often a tool for turning people against their own moral intuitions, especially in settings where people aren't able to encourage the behavior they actually want. For example, if you stopped people on the street ... (read more)

Fixed-weight models are adversarially vulnerable: hence misaligned

Stuart_Armstrong

This post argues that fixed-weight models (at least as we understand them today) will a) always be vulnerable to adversarial examples in their concept-spaces, and b) hence will be misaligned, under sufficient optimisation pressure.

Boundaries in concept space

To serve any purpose whatsoever, an AI will have to draw boundaries inside its world-model - to distinguish world A from world B, and reach some comparison between them.

If we want the AI to follow our goals and values, we want it to be able to recognise concepts like "human being", or maybe "conscious being", "suffering", "preference satisfaction", "law", and so on. So we want an AI to be able to look at a situation and assess, e.g., whether there are or aren't suffering conscious beings in it.

But concepts like "conscious...

(See More - 536 more words)

1jessicata6d

Mostly agree. 1. Yes I agree adversarial examples could mean the actor could exploit a stronger grader. 2. When you're talking about "rewarded" it seems we're back to the (1) case which isn't fixed-weights, unless by "rewarded" you mean something other than "RL trained". The interesting case is where the same weights are used for both action and grader, which adapt over time due to multi-agent RL, in which case it's like a game where the RL pseudo-Nash equilibrium is not obvious.

2Stuart_Armstrong5d

"rewarded" -> achieves a higher score or reward

jessicata5d32

I think the idea here is that deployment is like training, and during training it receives reward, and during deployment there may be epiphenomenal rewards even if they don't cause weight updates, and it's possible that "eval awareness" is low so it still pursues the epiphenomenal rewards?

I agree in this case, it is reasonably likely to seek reward during deployment, which includes tricking humans who grade it, however, this isn't optimizing on the concepts in its own weights, the issue is that it tricks humans. So it's not "misaligned because fixed weight... (read more)

2Stuart_Armstrong6d

Humans are adaptable enough to extend or modify their goals in many situations where these goals go "off distribution". The ACE algorithm I mentioned is capable of the same thing, in a much weaker form. Learning can allow entities to extend their goals intelligently to novel situations (rather than extending them blindly).

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Why I left Google DeepMind

Boundaries in concept space