AI ALIGNMENT FORUM
AF

AI Alignment Forum

20ryan_greenblatt

At the time when I first heard this agenda proposed, I was skeptical. I remain skeptical, especially about the technical work that has been done thus far on the agenda[1]. I think this post does a reasonable job of laying out the agenda and the key difficulties. However, when talking to Davidad in person, I've found that he often has more specific tricks and proposals than what was laid out in this post. I didn't find these tricks moved me very far, but I think they were helpful for understanding what is going on. This post and Davidad's agenda overall would benefit from having concrete examples of how the approach might work in various cases, or more discussion of what would be out of scope (and why this could be acceptable). For instance, how would you make a superhumanly efficient (ASI-designed) factory that produces robots while proving safety? How would you allow for AIs piloting household robots to do chores (or is this out of scope)? How would you allow for the AIs to produce software that people run on their computers or to design physical objects that get manufactured? Given that this proposal doesn't allow for safely automating safety research, my understanding is that it is supposed to be a stable end state. Correspondingly, it is important to know what Davidad thinks can and can't be done with this approach. My core disagreements are on the "Scientific Sufficiency Hypothesis" (particularly when considering computational constraints), "Model-Checking Feasibility Hypothesis" (and more generally on proving the relevant properties), and on the political feasibility of paying the needed tax even if the other components work out. It seems very implausible to me that making a sufficiently good simulation is as easy as building the Large Hadron Collider. I think the objection in this comment holds up (my understanding is Davidad would require that we formally verify everything on the computer).[2] As a concrete example, I found it quite implausible that you

4Charbel-Raphaël

Ok, time to review this post and assess the overall status of the project. Review of the post What i still appreciate about the post: I continue to appreciate its pedagogy, structure, and the general philosophy of taking a complex, lesser-known plan and helping it gain broader recognition. I'm still quite satisfied with the construction of the post—it's progressive and clearly distinguishes between what's important and what's not. I remember the first time I met Davidad. He sent me his previous post. I skimmed it for 15 minutes, didn't really understand it, and thought, "There's no way this is going to work." Then I reconsidered, thought about it more deeply, and realized there was something important here. Hopefully, this post succeeded in showing that there is indeed something worth exploring! I think such distillation and analysis are really important. I'm especially happy about the fact that we tried to elicit as much as we could from Davidad's model during our interactions, including his roadmap and some ideas of easy projects to get early empirical feedback on this proposal. Current Status of the Agenda. (I'm not the best person to write this, see this as an informal personal opinion) Overall, Davidad performed much better than expected with his new job as program director in ARIA and got funded 74M$ over 4 years. And I still think this is the only plan that could enable the creation of a very powerful AI capable of performing a true pivotal act to end the acute risk period, and I think this last part is the added value of this plan, especially in the sense that it could be done in a somewhat ethical/democratic way compared to other forms of pivotal acts. However, it's probably not going to happen in time. Are we on track? Weirdly, yes for the non-technical aspects, no for the technical ones? The post includes a roadmap with 4 stages, and we can check if we are on track. It seems to me that Davidad jumped directly to stage 3, without going through sta

Recent Discussion

Vanessa Kosoy's Shortform

Vanessa Kosoy

2Vanessa Kosoy3d

This is an idea I came up with and presented in the Agent Foundations 2025 at CMU conference. Here is a nice simple formalism for decision theory, that in particular supports the decision theory coming out of infra-Bayesianism. I now call the latter decision theory "Disambiguative Decision Theory", since the counterfactuals work by "disambiguating" the agent's belief. Formalism Let be the agent's event space and the space of possible policies [1] . Let be the agent's loss function. For each , we are given some . [2] This represents the event "the agent's behavior is consistent with policy ". We assume that This data is common for all decision theories, but the rest of the details depend on the theory: Functional Decision Theory (FDT) We are given a mapping . The distribution represents the logical counterfactual associated with . It is also possible to consider the more general "robust" version , but we will avoid it here for simplicity. The decision rule is then We will call an FDT problem "formally causal" when for any , the measures and agree when restricted to . That is, for any measurable , we require Causal Decision Theory (CDT) CDT has the same formal form as FDT, but we always require the problem to formally causal. Moreover, the interpretation of is different: it now represents the causal counterfactual associated with . The decision rule is also formally the same: Given an FDT problem , we can translate it to a CDT problem, if we specify the agent's belief about its own policies and causal interpretation: the kernel . Here is a copy of that represents the factual policy and is a copy of that represents the counterfactual policy. We require that , and that is formally causal in the second argument. Normally, comes from a causal graph, where apply the do-operator for the counterfactual policy and condition on the factual policy (i.e. condition on what the policy would have been if not for the do-operator). Given this data, we define th

Vanessa Kosoy9h20

A few more observations.

Partially Observable Iteration

The definition of iteration we had before implicitly assumes that the agent can observe the full outcome of previous iterations. We don't have to make this assumption. Instead, we can assume a set of possible observations and a mapping , in which case we define

I believe that Theorem 4 remains valid.

Idealized Disambiguative Decision Theory

As we remarked before, DDT is not invariant under adding a constant to the loss function. It is interesting to consider what happens when we add an increasingly large ... (read more)

Richard Ngo's Shortform

Richard_Ngo

9Daniel Kokotajlo4d

I don't think I understand how you are objecting to my point. Straightforwardly happy endings are the norm in fiction. I think you are reading too much into their choice to make the ending straightforwardly happy. Let's think about, idk, the Star Wars series for comparison. In it, the plucky heroes go to war against the bad guys, kill loads of them in a series of military engagements, and ultimately emerge victorious with their faction the new rulers of the galaxy. This is presented as straightforwardly happy ending. In fact lots of mainstream popular stories are of this pattern: The heroes defeat the bad guys in armed combat, kill loads of them, and then their faction has more political and military power afterwards and they all live happily and it's great and uncomplicated. Applying your reasoning to these popular stories, I guess you'd say: "Ominous. It seems like most Hollywood writers expect there to be a giant war in the future, possibly a civil war or series of assassinations. They are encouraging people to fight in this war / battle / revolution / assassination-series, and win, and they are portraying this as uncomplicatedly good." But I feel like there's a better explanation for what's going on: (a) Their audience likes fighting as a way of resolving conflicts, it's consistently entertaining and interesting to most people, so that's why so many movies conflicts resolve via combat, and (b) their endings are uncomplicatedly happy endings because again that's what most audiences want, that's the standard way to make a satisfying story.

6Richard_Ngo4d

I do actually think that the general trope of "the rebels winning is sufficient for a happy ending" is pretty indicative of poor ethical thinking. But even Hollywood balks at their heroes ending up with literal godlike control of the world. For example (though I haven't watched the series) my impression of the Avengers franchise is that they introduce a plot device (the infinity gauntlet) that gives its wielder godlike powers, the heroes use it specifically to defeat the bad guy and undo the damage he caused, and then they destroy the device. In other words, they got to the exact point that ratfic heroes got to, and then their happy ending specifically involves them giving up the same kind of godlike power that ratfic heroes typically use to make themselves dictators of the universe. Similarly for Superman: his happy endings involve him successfully using his godlike powers to beat the bad guys without changing the established world power structures basically at all. And I feel pretty confident that a big reason Superman doesn't end up taking over the world is because the writers and viewers would have moral qualms about that kind of ending. tl;dr: there are many ways to make a story have a happy ending, and it's quite indicative of the authors' ethical and political views which endings they consider to be happy. The kind of endings that rationalists often portray as happy, mainstream scriptwriters seem to go out of their way to avoid.

3Eli Tyre1d

Yes. And I claim they're wrong about that. There's lots of banal evil (some of which that is not regarded as evil by typical social morality, some of which is, but is generally treated as normal and ignored). I would fight a war to end factory farming, if that would help. If I ended up with "ultimate power" somehow, by some mechanism that didn't involve me taking on ultimate power for a specific narrow mandate, I think it is both ethically correct to use it to permanently end many (but probably not all) of those evils. This is indeed pretty scary.

Richard_Ngo17h20

Oh, I think of "ending factory farming" as very far from "taking over the world".

If Superman were a skilled political operator it could be as simple as arranging to take photoshoots with whichever politicians legislated the end of factory farms.

Or if he were less skilled it could involve doing various kinds of property damage to factory farms (potentially even things which there aren't laws against, like flying around them in a way which blows the buildings over).

This might escalate to the government trying to arrest him, and outright conflict, but honestl... (read more)

[Intro to brain-like-AGI safety] 6. Big picture of motivation, decision-making, and RL

Steven Byrnes

(Last revised: January 2026. See changelog at the bottom.)

6.1 Post summary / Table of contents

Part of the “Intro to brain-like-AGI safety” post series.

Thus far in the series, Post #1 set out some definitions and motivations (what is “brain-like AGI safety” and why should we care?), and Posts #2 & #3 split the brain into a Learning Subsystem (cortex, striatum, cerebellum, amygdala, etc.) that “learns from scratch” using learning algorithms, and a Steering Subsystem (hypothalamus, brainstem, etc.) that is mostly genetically-hardwired and executes innate species-specific instincts and reactions.

Then in Post #4, I talked about the “short-term predictor”, a circuit which learns, via supervised learning, to predict a signal in advance of its arrival, but only by perhaps a fraction of a second. Post #5 then argued that if we form a closed...

(Continue Reading - 5056 more words)

Linda Linsefors1d20

The big picture—The whole post will revolve around this diagram. Note that I’m oversimplifying in various ways, including in the bracketed neuroanatomy labels.

I think this picture would be clearer if you drew [predict sensory inputs] as a separate box from Though Generator.

In the picture in my head, there is [predict sensory inputs] box, that revives and tries to predict the sensory output. This box also sends a signal of [current context] to both the Though Generator and the Though Assessor. Also, [predict sensory inputs] gets some signal from Though Ge... (read more)

Schelling Goodness, and Shared Morality as a Goal

Andrew_Critch

1mo

Also available in markdown at theMultiplicity.ai/blog/schelling-goodness.

This post explores a notion I'll call Schelling goodness. Claims of Schelling goodness are not first-order moral verdicts like "X is good" or "X is bad." They are claims about a class of hypothetical coordination games in the sense of Thomas Schelling, where the task being coordinated on is a moral verdict. In each such game, participants aim to give the same response regarding a moral question, by reasoning about what a very diverse population of intelligent beings would converge on, using only broadly shared constraints: common knowledge of the question at hand, and background knowledge from the survival and growth pressures that shape successful civilizations. Unlike many Schelling coordination games, we'll be focused on scenarios with no shared history or knowledge...

(Continue Reading - 11999 more words)

Andrew_Critch2d30

Which universal distribution?

Some universal distributions are full of agents that make choices that make that distribution not a valid model of reality after the decisions are made (self-defeating). Other distributions are full of agents making decisions that ratify the distribution (self-fulfilling).

Distributions that aren't fixed points under reflection about what they decide about themselves are not coherent models of reality.

Buck's Shortform

Buck

20Buck2d

Rob Wiblin asked: [...] My response: [...] As additional context, I'll note that I was just talking about misalignment risk and security; I didn't mention considerations relevant to e.g. preventing coups.

4Raemon2d

The "Anthropic might actually be worst as an epistemic environment for safety arguments" rhymes with stuff I've worried about. I am curious, if you were Anthropic leadership or (a few) high profile employees, if you think there are things they could steer towards that'd help with this problem, that are, like, incentive-compatible with Anthropic's overall mission/goals/psychology.

5Lukas Finnveden2d

In what sense are OAI's models more CoT-monitorable?

Buck2d83

OAI models rely more on CoT for their capabilities. E.g. their benchmark scores with and without CoT are more different.
Anthropic models treat their CoT less differently from their output than OAI models do. This means that RL probably pressures their CoT more. See here.

Alex Mallen's Shortform

Alex Mallen

9mo

3Linch2d

I'm confused. Can someone explain to me in simple language why an RL environment for twin-prisoner's dilemmas wouldn't favor EDT?

Alex Mallen2d20

Let's say the current policy has a 90% chance of cooperating. Then, what action results in the highest expected reward for player 1 (and in turn, gets reinforced the most on average)? Player 1 sampling defect leads to a higher reward for player 1 whether or not player 2 samples cooperate (strategic dominance), and there's a 90% chance of player 2 sampling cooperate regardless of player 1's action because the policy is fixed (i.e., player 1 cooperating is no evidence of player 2 cooperating, so it's not the case that reward tends to be higher for player 1 w... (read more)

1Alex Mallen4d

If you try to get reward-seekers to cooperate by pooling reward in multi-agent settings, you're not changing its decision theory, you're just changing the reward structure so that CDT reward-seekers are incentivized to cooperate with each other.

3Buck4d

Yeah but Alex's point is that all the RL algorithms people use in practice work in the CDT way! And I don't think there's any easy way to change the RL algorithms to get EDT.

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Partially Observable Iteration

Idealized Disambiguative Decision Theory

6.1 Post summary / Table of contents