To understand why powerful AIs might get into conflict, and ways to mitigate it, we need to understand bargaining problems: situations where multiple agents have different preferences over Pareto-efficient outcomes.

I’ve come to suspect that certain common frames on bargaining problems are confused. Here, I’ll explain why, and which frames I think are better. One motivation for this is to hopefully help others make progress in research on safe Pareto improvements (SPIs), which are among the most promising approaches to mitigating the downsides of AI conflict, in my view.

In this post I’ll:

give some relevant background from my previous writings; (more)
introduce mechanistic explanations as a crucial frame for AI bargaining research, using, as a running example, explanations of failure to coordinate demands in bargaining (this example also motivates the following bullets’ claims); (more)
argue that instead of asking what would happen in Nash equilibrium, we should ask what kinds of beliefs AI bargainers would plausibly have about each other; (more) and
argue that instead of relying much on heuristics about (e.g.) what agents would consider “exploitative”, we should carefully spell out which commitments they expect to have good consequences. (more)

Background

I recommend reading the following beforehand (at least the linked sections), but it’s not strictly required, since the essential parts for our purposes are in the quotes below.

“A high-level model of AI bargaining”, which spells out what kinds of strategic situations we’re aiming to understand:

“[‘Bargaining’ between agents Alice and Bob] consists in credibly reporting to each other their (i) demands/offers and (ii) policies for which outside options they’d each take[, i.e., what they’d do] if bargaining failed (such as leaving the resource alone, or initiating conflict).”
“Each agent’s procedure for [how they’ll bargain], called a program, takes as input information about the other agent’s program, as well as other features of the strategic situation. … So, the AIs can implement conditional commitments, instead of necessarily either locking in rigid demands or conceding to whoever commits first.”
“[Before bargaining, each agent can] open or close information channels … through which they learn whether the other agent has made some credible commitment.”

“Responses to apparent rationalist confusions about game / decision theory”:

“Ex post optimal =/= ex ante optimal”: “Each of us is incentivized to choose our demand without knowing what exactly the other will demand, because if you wait to eliminate your uncertainty before making a demand, you lose the opportunity to influence the bargain with your commitment.”

Mechanistic explanations of miscoordination in bargaining problems

If we want to make reasonable (even if coarse-grained) predictions about how AI bargaining will work, we should aim to mechanistically explain ^[1] bargaining behavior. This may sound trite. But in my experience, it’s easy to neglect mechanistic explanations and fall back on models or heuristics that don’t fit this context.

To see what I mean, we’ll start with the central question about AI bargaining:

We’re supposing that our AI bargainers Alice and Bob are technologically advanced enough to make fully credible, conditional commitments (via programs). And they can overcome private information problems. Then, why wouldn’t they always coordinate on compatible demands?

I’ve partly answered this with the “Ex post optimal =/= ex ante optimal” quote above. ^[2] But that argument is imprecise. For example, what does “influence” mean here? What kinds of “waiting to eliminate your uncertainty” are relevant? Likewise, we might answer the central question by appealing to “equilibrium selection”. This gestures at the problem, but doesn’t really explain why agents with conditional commitment abilities would fail to coordinate.

Instead, we should get clear on what mechanistically explains miscoordination — because these explanations will inform our predictions about other aspects of AI bargaining, e.g., when agents will individually prefer to use safe Pareto improvements (SPIs). So let’s break down different ways we might think Alice and Bob can guarantee their demands are coordinated, and how each of them can fail. (This is a weaker claim than “miscoordination of demands is inevitable or rational” or “risking costly conflict is inevitable or rational”, both of which seem false.)

Option 1: Bob waits to observe Alice’s program

One way for Bob to make conflict less likely is to condition his program choice on Alice’s demands, that is: before choosing his program, first learn which demands Alice’s program is likely to make against different programs (e.g., by accessing a trusted server where Alice’s program is registered), then update on this information. ^[3] If Bob does this, he’s more likely to make demands compatible with Alice’s.

But suppose Alice’s program can make different demands depending on whether Bob accessed the server. Then:

Bob knows that Alice has an incentive to put this clause in her program: “If Bob accessed the server before locking in his own program, then demand more of the resource than otherwise.”
- Why? Because she might think, “If Bob looked at the server, probably the reason he did so was to condition his program choice on my demands. And if he conditions his program choice on my demands, that’s some evidence that his program will concede to those demands. Then I’m better off demanding more.” ^[4]
So, given his beliefs, Bob faces the choice between:
- (a) looking first, thereby eliminating the risk of conflict but making it more likely he gets a worse deal; and
- (b) not looking, thereby risking conflict but getting a better deal if Alice agrees.
Depending on Bob’s values, risk tolerance, and beliefs about the likelihood of winning in the conflict, he might prefer (b), and commit before learning Alice’s program.

(This is the dynamic captured by the “partial answer” above.)

Option 2: Bob conditions his demands on Alice’s program

Bob now wonders which program he should blindly choose. He considers a program of this general form: “Make my initial demands with some ‘first-order’ program. Check if the demands made by the first-order program and Alice’s program against each other would be incompatible. If so, make lower demands than the first-order program’s demands.” ^[5] We’ll say a program of this form conditions its demands on incompatibility. This kind of program, again, would be more likely to make compatible demands with Alice’s.

But Bob faces the same problem as before! Alice has an incentive to give her program the clause: “If Bob’s program conditions its demands on incompatibility, then demand more than otherwise.” So if he’s willing to accept the risk, he might commit to a program that doesn’t condition its demands on incompatibility.

Option 3: Alice and Bob share a decision procedure

Finally, even if Options 1 and 2 fail, perhaps Alice’s and Bob’s demands can be acausally coordinated? The idea is: Suppose Alice and Bob follow acausal decision theories. Then, they each might regard their choice as determining the other’s even without communicating, if they think they make decisions according to sufficiently similar procedures.

For our purposes, let’s say an agent’s decision procedure is the process that generates their choice of program given their evidence. To quote an objection addressed by the “Rationalist confusions” post:

Virtually all agents who are sufficiently capable to enter high-stakes bargaining interactions will converge on the same decision procedure, and reason that their decision to [commit to a fair demand] logically causes their counterparts to do likewise [or is strong evidence that they’ll do likewise].

In that same section of the post, I explain why this, too, isn’t guaranteed. This quote captures most of the problem (footnote added):

If the reason to expect convergence of decision procedures in bargaining problems is that we expect selection for the same decision theory, then … merely sharing the same decision theory that you consult as an ideal doesn’t mean you’ll share a whole decision procedure (with respect to the given problem). For example, you might not share how you model the decision problem, what kinds of evidence about other agents are most salient to you, how you approximate ideal Bayesianism, etc. ^[6]

From mechanistic explanations to better research frames

So, we’ve got a mechanistic story for why AI bargainers might not coordinate on compatible demands: Bob anticipates that Alice has an incentive to use a program that bargains harder if he tries to avoid incompatible demands — either via conditioning his program choice on her demands, or conditioning his demands on incompatibility. (And vice versa.) And there aren’t sufficiently strong selection pressures toward very similar decision procedures.

But what does all this have to do with how we should think about AI bargaining research, more broadly? Here are two general implications.

Constraints on beliefs, rather than equilibrium

Standard game theory, including lots of bargaining theory, predicts that rational agents will play a Nash equilibrium, where each agent’s strategy is optimal against the others’ actual strategies.

But in the kinds of bargaining problems we’re focusing on here, this is a very strong condition on its face! There’s a wide range of possible conditional commitments the AIs might implement. So realistically, AIs will have nontrivial uncertainty about each other’s strategies.

And here’s the payoff of the previous section: There’s an incentive not to resolve this uncertainty before locking in one’s program. That is, if an agent gains enough information about their counterpart for the prediction of equilibrium to make sense, this could leave them in a worse equilibrium (for themselves). I’m not aware of any standard justifications for predicting Nash equilibrium that explicitly acknowledge this kind of incentive. This should make us suspicious of such predictions in the AI bargaining context specifically, even if we’re inclined to defer to the equilibrium-first framework in general because it’s very well-established. (For what it’s worth, skepticism about equilibrium has academic precedent in epistemic game theory (h/t Jesse Clifton). ^[7] ^[8] )

Now, “nontrivial uncertainty” alone doesn’t imply the AIs won’t be in equilibrium. Shouldn’t we still expect advanced AIs to have beliefs about each other that are reasonable enough to avoid miscoordination? ^[9] I’m not sure we should. Here are some defenses of that view and why I don’t find them convincing:

Objection: “Rational agents will generally have well-calibrated beliefs. Even without directly observing each other’s programs, they can learn a lot from each other’s past behavior, and from reasoning carefully about the sorts of mechanistic explanations you’re advocating. (There are also formal results in this direction, like Kalai and Lehrer (1993), ‘Rational Learning Leads to Nash Equilibrium’, and Mailath (1998), ‘Do People Play Nash Equilibrium?’ (Sec. 4).)”
- Response: What matters are the agents’ beliefs about each other’s strategies in the given interaction. And the problem is that these beliefs seem underdetermined by the data the AIs would have about each other, plus any constraints on “reasonable” priors. For instance, suppose Alice tries to forecast Bob’s program choice based on his past decisions in multi-agent contexts. Bob made such decisions based on his beliefs about others’ strategic reasoning. But he has presumably updated these beliefs over time and across contexts, while trying to make his own strategic forecasts. Not to mention, all of these interactions might have been much lower-stakes, or under less anarchic conditions, than the bargaining problem Alice and Bob now face. So there seems to be a distributional shift problem. How exactly should Alice extrapolate from this data? It’s unclear.
Objection: “Rational agents should predict that other rational agents reason similarly to them. So they can accurately predict other rational agents’ strategies by introspecting on what kinds of considerations are salient to themselves.”
- Response: It seems implausible to model all “rational” agents as able to very accurately predict each other. More realistically, a variety of agents will reason in different ways consistent with avoiding clear failures of rationality. If so, a better model would be: There are several clusters of rational agents similar to each other in more fine-grained ways, and only agents within a given cluster can accurately predict each other based on similarity alone.
Objection: “Empirically, agents’ strategic behavior does seem to adhere to the equilibrium model.”
- Response: This seems false, or at the very least it’s a strong claim that needs positive substantiation. Humans routinely deviate from equilibrium models, even when the space of strategies is fairly simple compared to the space of conditional commitments AIs might reason over.

So I remain skeptical that we should expect AI bargainers to play Nash equilibria. But we might ask, what else is there?

I think the way forward is to stop looking for game-theoretic “solution concepts” per se. Instead, we should model the agents as individuals making decisions under uncertainty, and see which strategic dynamics follow from plausible properties of such decision-making. Here’s an eloquent statement of this frame:

There is no special concept of rationality for decision making in a situation where the outcomes depend on the actions of more than one agent. The acts of other agents are, like chance events, natural disasters and acts of God, just facts about an uncertain world that agents have beliefs and degrees of belief about. The utilities of other agents are relevant to an agent only as information that, together with beliefs about the rationality of those agents, helps to predict their actions. (Stalnaker 1996: 136)

That is, if we’re going to make predictions about agents’ strategic decisions, I suggest we look at what kinds of beliefs they might have about each other, and why. This is hard, of course. But retreating to Nash because of theoretical tractability would be looking under the streetlight. And I think we can at least say some coarse-grained things about what kinds of beliefs are plausible. Examples of this methodology:

The analysis of options 1 and 2 above.
Reasoning about the features of strategies that make them focal/Schelling points.
Modeling selection pressures that favor certain kinds of beliefs or strategies.
The “Belief assumptions” bullet here, which gives a prima facie argument for particular constraints on agents’ beliefs. If these constraints hold, we can bound the losses from conflict in bargaining between agents in a certain class of games. (This argument faces some critiques; see the sub-bullet of “Belief assumptions”. But I see these critiques, too, as demonstrations of my preferred methodology.)

Analysis of expected consequences, rather than (unscrutinized) heuristics

Recall the arguments in the “Mechanistic explanations of miscoordination” section. How did these arguments work? Well, we considered some actions Bob might take, asked what Bob might believe about how Alice’s program would likely respond to each action (and why he’d believe this), and checked which action Bob would expect to do best given these beliefs. By “expect to do best”, I mean to include various forms of weighing up actions’ possible consequences by their value and likelihood, not just literal expected utility maximization.

That’s it. We didn’t say anything like:

“Bob doesn’t want to be exploited.”
“If Alice committed to unfair demands, Bob would want to punish this.”
“By committing blindly, Bob is trying to deter Alice from reducing his utility.”
“Bob reasons as if he moves first (perhaps first in ‘logical time’).”

These sorts of claims are heuristics, not mechanistic explanations — at least, not without saying more about why we should expect Bob to reason in these ways. ^[10] They don’t tell us precisely why Bob might not (say) condition his demands on incompatibility.

In principle, heuristics can capture considerations our current mechanistic models leave out. But we at least need to point to reasons to trust that a given heuristic really is tracking such considerations.

And obviously, sometimes imprecise shorthand is convenient. For instance, I find the concept of “bargaining power” helpful for quickly conveying some intuitions here. But we’ll get confused if we take the shorthand’s connotations too literally when working out its implications — e.g., predicting that any opportunity to mitigate conflict comes at the expense of “bargaining power” (SPIs are a counterexample!). Disambiguating heuristics about bargaining helps us avoid talking past each other, and shows us when the implications we’re drawing from heuristics don’t actually seem to follow from (good) mechanistic explanations.

As an example, ^[11] imagine Alice commits to participate in some SPI with Bob if his program meets certain conditions (the details of which don’t matter here). Bob doesn’t know this yet, since he has stayed blind to all of Alice’s strategic decisions thus far. Now suppose we explain bargaining problems by saying that Bob “reasons as if he moves first”. We might end up concluding: “Bob might want to commit to a program that ignores all offers to participate in SPIs. After all, if he conditioned his program’s output on such an offer, he’d be moving second, which is risky/exploitable ex ante.”

I don’t find such an argument compelling on its own. It’s possible that Bob expects to do worse purely by conditioning on information about Alice’s program, no matter what specific kind of information it is. Yet it’s not clear why exactly he’d be likely to expect this. Talking about “moving second” doesn’t tell us that.

Some examples of analyzing expected consequences in multi-agent interactions, without relying on unscrutinized heuristics:

Fearon (1995), “Rationalist Explanations for War”, which mechanistically explains conflict in terms of (among other things) risk-reward tradeoffs under uncertainty about private information.
The discussion here (section “Unexploitable open-mindedness”) of how avoiding “exploitation” can come apart from ex ante optimality.
This analysis (section “Why developers are better suited”) of how AI bargainers’ predecessors might have an incentive to coordinate even if the AIs themselves might not.

Acknowledgments

Thanks to John Massey, Tristan Cook, Lukas Finnveden, Cadence James, Matt Hampton, and Paul Knott for helpful comments.

Note that “mechanistically” doesn’t have anything to do with mechanism design. ↩︎
This dynamic might remind you of the “commitment races” problem. I think the commitment races frame can be helpful in some contexts. But it’s also loaded with associations that my argument here isn’t committed to (pun not intended), and I’m not aware of public writings precisely spelling out the argument in the rest of this section. ↩︎
Or, if he follows something like updateless decision theory, then he’s deciding whether to act as if updating on this information. ↩︎
Commentary on some other possibilities here: (i) Alice might also be capable of conditioning her program’s demands directly on whether Bob conditions his program choice on her demands, rather than on the crude proxy “did Bob look first?”. If so, this would presumably be a more robust option than using the crude proxy. (ii) Suppose Alice can’t design a program that conditions its demands on whether Bob looked first at all. Here’s another thing she could do with the same strategic consequences: monitor for whether, and when, Bob accesses the server — once he has done so, lock in a program that always makes high demands. ↩︎
If you’re familiar with “renegotiation programs”, this might look familiar. But renegotiation programs are importantly different. First, a renegotiation program rn(p) doesn’t adjust to the counterpart unconditionally: It only behaves differently from p if, against p, the counterpart’s program would’ve made the same demands as its own first-order program would. So the failure mode in Option 2 doesn’t apply to renegotiation programs, because a counterpart who demands more against rn(p) than against p thereby disqualifies themselves from the renegotiation. Second, in the case where rn(p) “behaves differently” from p, it doesn’t necessarily fall back to lower demands in particular. ↩︎
Something I left out of that post: Even if the agents share all these things, they might also not have access to the same evidence, when they decide what programs to commit to. ↩︎
Bicchieri (1995) (emphasis mine): “Yet even when common knowledge is present, there are games in which this much knowledge is not sufficient to infer a correct prediction of the opponents’ choices. Since playing a Nash equilibrium involves correct expectations on the part of the players, one might impose the additional epistemic requirement that the beliefs that players hold about the strategies of others are common knowledge. I shall argue that common knowledge of beliefs is an implausible assumption. Moreover, the assumption does not guarantee that players will end up with correct expectations if they have mutually inconsistent beliefs to begin with.” ↩︎
Pearce (1984) (emphasis mine): “The most sweeping (and, perhaps, historically the most frequently invoked) case for Nash equilibrium theory in such circumstances asserts that a player’s strategy must be a best response to those selected by other players, because he can deduce what those strategies are. Player i can figure out j’s strategic choice by merely imagining himself in j’s position. But this takes for granted that there is a unique rational choice for j to make; this uniqueness is not derived from fundamental rationality postulates, but is simply assumed. Furthermore, any argument suggesting that player rationality, combined with the structural characteristics of a game, inevitably renders all but one outcome ‘impossible,’ leads to conclusions that contradict widely accepted notions of ‘perfection’ (Pearce [19]). Once one admits the possibility that a player may have several strategies that he could reasonably use, expectations may be mismatched. Player i’s strategy will then be a best response to his (possibly incorrect) conjecture about others’ strategies, not the actual strategies employed. … [W]e are interested in analyzing many situations for which no precedents exist (such as nuclear wars between superpowers) or in which continual changes in relevant variables (technological breakthroughs, new legislation, and so on) preclude prediction based on tradition. It then becomes crucial to understand precisely what are the implications of players’ information and rationality.” ↩︎
Separately, if you’re familiar with Bayesian Nash equilibrium, you might be wondering why we don’t just use that instead. That is, we could model the agents as (a) uncertain of each other’s private information, but (b) still best-responding to each other’s function from that private information to actions in the game. I think this is an improvement. But this also isn’t enough, because we want to represent agents’ uncertainty about each other’s choices of strategies/programs, and private information isn’t strategically chosen. ↩︎
For example, if humans and current AIs find some crude strategic heuristics intuitive, that might be evidence that future AI bargainers will, too. I find it hard to say how strong this evidence is. But I think we should be explicit about when we’re making this kind of inference, and not conflate it with an argument from “rational” incentives alone. ↩︎
I made this up for illustration; it’s not from any actual researcher as far as I know, though it’s somewhat related to Soto’s discussion here. The real examples take more context and unpacking than makes sense for a post of this scope. ↩︎