This is a pared-down version of a longer draft report. We went with a more concise version to get it out faster, so it ended up being more of an overview of definitions and concepts, and is thin on concrete examples and details. Hopefully subsequent work will help fill those gaps.

Sequence summary^[1]

Some researchers are focused on reducing the risks of conflict between AGIs. In this sequence, we’ll present several necessary conditions for technical work on AGI conflict reduction to be effective, and survey circumstances under which these conditions hold. We’ll also share some thoughts on promising directions for research and intervention to prevent AGI conflict.

This post
1. We give a breakdown of necessary conditions for technical work on AGI conflict reduction to make a difference: “AGIs won’t always avoid conflict, despite it being materially costly” and “intent alignment is either insufficient or unnecessary for conflict reduction work to make a difference.” (more)
When would AGIs engage in conflict?
1. To assess the claim that AGIs would figure out how to avoid conflict, we give a breakdown (which we believe is exhaustive) of the causes of conflict between rational agents. Some kinds of conflict can be avoided if the agents are sufficiently capable of credible commitment and disclosure of private information. Two barriers to preventing conflict by these means are (i) strategic pressures early in multipolar AI takeoff that make it risky to implement cooperation-improving technologies, and (ii) fundamental computational hurdles to credible commitment and disclosure. Other causes of conflict — miscoordination and conflict-seeking preferences — can’t be solved by credible commitment and information disclosure alone. (more)
2. We give examples of technical work aimed at each of the causes of conflict surveyed in (2a). (more)
When is intent alignment sufficient or necessary to reduce AGI conflict?
1. We survey conditions under which intent-aligned AGIs could engage in conflict that is not endorsed by their overseers. Scenarios include: locking in architectures that can’t solve informational and commitment problems; the unintentional corruption of human preferences by AGIs; and catastrophes which leave human overseers unable to prevent intent-aligned AGIs from acting on misunderstandings of their preferences. (more)
2. We consider whether it is pos

...

When does technical work to reduce AGI conflict make a difference?