This is a special post for quick takes by TurnTrout. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I regret each of the thousands of hours I spent on my power-seeking theorems, and sometimes fantasize about retracting one or both papers "Optimal policies tend to seek power." I am pained every time someone cites "Optimal policies tend to seek power", and despair that it is included in the alignment 201 curriculum. I think this work makes readers actively worse at thinking about realistic trained systems.
I think a healthy alignment community would have rebuked me for that line of research, but sadly I only remember about two people objecting that "optimality" is a horrible way of understanding trained policies.
EDIT Oct 7, 2026: I think, in my despair of the "optimality" mental trap, I went too far with this comment. I likely subconsciously exaggerated my feelings to increase rhetorical impact ( :S ). I do not, at this point, regret each of the thousands of hours I spent on the theorems, and probably I didn't fully regret it at the time of writing.
I think the power-seeking theorems are cool, especially the ones that generalize beyond optimality.
I think my research contributed to confusion around "optimality" (with respect to reward functions).
I think that instrumental convergence is worth thinking about.
I think my formalizations of power, goal- and environment-symmetry, and retargetability comprised significant conceptual advances.
I think, in my despair of the "optimality" mental trap, I went too far with the parent comment. I likely subconsciously exaggerated my feelings to increase rhetorical impact ( :S ). I do not, at this point, regret each of the thousands of hours I spent on the theorems, and probably I didn't fully regret it at the time of writing.
I think the power-seeking theorems are cool, especially the ones that generalize beyond optimality.
I think my research contributed to confusion around "optimality" (with respect to reward functions).
Since I'm an author on that paper, I wanted to clarify my position here. My perspective is basically the same as Steven's: there's a straightforward conceptual argument that goal-directedness leads to convergent instrumental subgoals, this is an important part of the AI risk argument, and the argument gains much more legitimacy and slightly more confidence in correctness by being formalized in a peer-reviewed paper.
I also think this has basically always been my attitude towards this paper. In particular, I don't think I ever thought of this paper as providing any evidence about whether realistic trained systems would be goal-directed.
Just to check that I wasn't falling prey to hindsight bias, I looked through our Slack history. Most of it is about the technical details of the results, so not very informative, but the few conversations on higher-level discussion I think overall support this picture. E.g. here are some quotes (only things I said):
Nov 3, 2019:
[...]
Dec 11, 2019:
[...]
3TurnTrout
I think this is reasonable, although I might say "suggesting" instead of "showing." I think I might also be more cautious about further inferences which people might make from this -- like I think a bunch of the algorithms I proved things about are importantly unrealistic. But the sentence itself seems fine, at first pass.