Much informal discussion of the Unilateralist’s Curse from Bostrom et al (2016) presents it as a sort of collective action problem: the chance of purely altruistic agents causing harm rises with the number of agents that act alone. What’s often left out is that these agents are irrational. The central result depends on this, and I’ll show why below.

Note that the formal result in the original paper is correct. The authors are largely aware of what I’ll explain here; they discuss much of it in section 3.2. The point of this post is (i) to correct a misconception about which agents the Curse applies to, (ii) to go through a particularly neat part of the game theory canon, and (iii) to highlight some nuances.

Summary

Even when agents have no information about what others think, it is not generally an equilibrium for them to act merely based on their own all-things-considered belief.
In equilibrium, rational and altruistic agents may or may not converge (in the limit) to avoiding the kind of mistake highlighted by the Unilateralist's Curse.
Rational and altruistic agents can still do far better by using, e.g., majority vote.
Empirical results suggest that agents do behave quite rationally in various contexts.

I suggest skipping the derivations if you're not interested.

Why Naïve Unilateralism Is Not an Equilibrium

The main theoretical insights I’ll explain are from Feddersen and Pesendorfer (1998).^[1] I simply apply them to the case of the Unilateralist’s Curse. I’ll illustrate using the first example from Bostrom et al (2016, p. 1) where they motivate the issue:

A group of scientists working on the development of an HIV vaccine has accidentally created an air-transmissible variant of HIV. The scientists must decide whether to publish their discovery, knowing that it might be used to create a devastating biological weapon, but also that it could help those who hope to develop defenses against such weapons. Most members of the group think publication is too risky, but one disagrees. He mentions the discovery at a conference, and soon the details are widely known.

Next, a formal setup:^[2]

There are agents.
An agent can either write a paper ( $w$ ) or stay silent ( $s$ ).
If at least one agent writes the paper, it is published ( $W$ ); if none do, it is not ( $S$ ).
Having the paper written would in fact either be good ( $G$ ) or bad ( $B$ ) on net. For simplicity, the common prior is $P (G) = P (B) = \frac{1}{2}$ .^[3]
Each agent $i$ gets an informative i.i.d. signal $m_{i} \in {g, b}$ , i.e. their own impression, about the effect of such a paper being written. For simplicity, it’s symmetric: $p ≜ P (m_{i} = g | G) = P (m_{i} = b | B) > \frac{1}{2}$ for any $i \in N_{n}$ .
The agents are equally altruistic: $u_{i} = 1_{{W, G}} + 1_{{S, B}} \forall i \in N_{n}$ . An error gives all agents a payoff of $0$ while a correct collective decision gives each $1$ .^[4]

The solution concept is Bayesian Nash Equilibrium.^[5]

Claim: The following is not an equilibrium: agents choose to write the paper just in case their own impression suggests it would be good.

Proof. Suppose for contradiction that this is an equilibrium. It’s symmetric so consider any agent $i$ . By assumption, $i$ only cares about the outcome. If at least one other agent writes the paper, then $i$ can’t affect the outcome. If nobody has written the paper, $i$ can unilaterally pick the outcome. Since $i$ ’s choice makes no difference to their payoff when someone else has chosen to write the paper, $i$ will simply condition on nobody else having chosen to write the paper. (For the unconvinced: ^[6] .)

Denote by $D$ the 'decisive' case where nobody else has chosen to write the paper. Now that $i$ conditions on being decisive, $n - 1$ agents must have chosen to stay silent. By supposition, this implies that $n - 1$ agents got signal $b$ . Suppose that $i$ got signal $g$ . By conditioning on being decisive, $i$ ’s relevant posterior becomes

P (G | D, m_{i} = g) = \frac{P (D, m_{i} = g | G) P (G)}{P (D, m_{i} = g | G) P (G) + P (D, m_{i} = g | B) P (B)}

= \frac{(1 - p)^{n - 1} p \frac{1}{2}}{(1 - p)^{n - 1} p \frac{1}{2} + p^{n - 1} (1 - p) \frac{1}{2}} = \frac{1}{1 + p^{n - 2} (1 - p)^{1 - (n - 1)}} = \frac{1}{1 + (\frac{p}{1 - p})^{n - 2}}

which is always less than 50%.^[7]

Therefore, $i$ will act as if writing the paper is likely harmful. So $i$ won’t write the paper—even though (i) their own impression suggests it’d be good and (ii) they have no information about what others think. This contradicts the supposition, proving the claim.

What happened to the Unilateralist's Curse? The authors were careful to write "if each agent acts on her own personal judgment" in the paper's abstract because this is a nontrivial condition. It requires that agents observe their signal and naïvely act upon it. But as we've shown, if all agents follow this rule, then writing the paper becomes a strictly inferior act. A rational agent would not write.

Here's the idea in stream-of-consciousness: I think it's worth writing the paper. Of course, it doesn't matter whether I write it if someone else will anyway. But if nobody else does, then I could do some good. Wait. If nobody else would write the paper, then they must all think it's a bad idea. I'm only as sensible as they are, so I must presumably have missed something. I guess I'll stay silent then.

That's all well and good for intuiting why rational unilateralists don't just follow their signal. But did you spot the mistake in that reasoning?

The stream of consciousness should have continued: Interesting, it seems like the best move is to stay silent no matter what I personally think. Hold on! The other agents must've thought of this too, implying that they'd just stay silent no matter what. But now being decisive tells me nothing at all about what the others think! So I'll just use my own judgment and write the paper. Wait...

And so forth.

What, then, does a rational agent do in this case? Does this game just not have an equilibrium? It's got finite players and finite pure strategies, so by a simple extension of Nash's existence theorem, there must be an equilibrium.

Deriving Equilibria

There are actually multiple equilibria. I'll only describe a nontrivial symmetric one here. Let's see whether we can find an equilibrium in which agents stay silent when their own impression is that publication would be bad. We then just need to specify what they would do when their own impression is that writing the paper would be beneficial. Given the discussion above, it won't simply be "write" or "don't write." If there's to be an equilibrium, it will have to be in mixed strategies—where agents randomise.

Let's complicate the setup a bit to see some interesting dynamics:

Now $u_{i} = - q 1_{{W, B}} - (1 - q) 1_{{S, G}} \forall i \in N_{n}$ for some $q \in (0, 1)$ . This normalises the payoff of a correct decision to $0$ . Payoffs for the wrong decisions are negative.
We define $p_{g}$ as the chance of an agent with signal $g$ writing a paper.

Begin by noticing that $q$ can be thought of as the probability threshold at which an agent would want the paper to be written. To see this, observe that the expected payoffs are

$E [u_{i} (W)] = P (G) \times 0 + [1 - P (G)] (- q)$ and
$E [u_{i} (S)] = P (G) [- (1 - q)] + [1 - P (G)] \times 0$ ,

so $i$ favours publication whenever $- q [1 - P (G)] > - (1 - q) P (G) ⟹ P (G) > q$ .^[8]

For an agent to write the paper with a probability other than zero or one, they must be indifferent between writing and not writing it.^[9] That is, $P (G) = q$ . Like before, an agent's choice only matters when they are decisive. So we need to find a $p_{g}$ satisfying

P (G | D, m_{i} = g) = \frac{P (D, m_{i} = g | G) P (G)}{P (D, m_{i} = g | G) P (G) + P (D, m_{i} = g | B) P (B)}

= \frac{\frac{1}{2} P (m_{i} = g | G) [P (s | m_{j} = b) P (m_{j} = b | G) + P (s | m_{j} = g) P (m_{j} = g | G)]^{n - 1}}{P (D, m_{i} = g | G) \frac{1}{2} + P (D, m_{i} = g | B) \frac{1}{2}}

= \frac{p [(1 - p) + (1 - p_{g}) p]^{n - 1}}{p [(1 - p) + (1 - p_{g}) p]^{n - 1} + (1 - p) [p + (1 - p_{g}) (1 - p)]^{n - 1}}

= {⎛ ⎝ 1 + \frac{1 - p}{p} {[\frac{1 - p_{g} (1 - p)}{1 - p_{g} p}]}^{n - 1} ⎞ ⎠}^{- 1} = q .

Re-arranging and bounding appropriately gives us

p_{g} = min {1, max {0, p_{g}^{'}}} where p_{g}^{'} = \frac{{[\frac{(1 - q) p}{q (1 - p)}]}^{\frac{1}{n - 1}} - 1}{p (1 + {[\frac{(1 - q) p}{q (1 - p)}]}^{\frac{1}{n - 1}}) - 1} .

Finding $p_{g}$ is just a matter of plugging in values for $p$ , $q$ , and $n$ .^[10] For instance, if 15 agents all have a probability threshold of 50% and their impressions are 75% accurate, then the chance of any individual with a positive impression writing the paper is $p_{g} \approx 15 %$ .

More interestingly, though, we can examine the comparative statics. Let's start with the limiting behaviour as the number of rational, altruistic agents rises:

We always have ${lim}_{n \to \infty} p_{g} (n) = 0$ .^[11] The more agents there are, the lower the chance that any particular individual with a positive impression writes the paper.
And yet ${lim}_{n \to \infty} P (W | B) \notin {0, 1}$ in general.^[12] As the number of agents rises, the probability that the paper is written at all when it shouldn't be converges to some constant, determined by both the accuracy and threshold.
Similarly, ${lim}_{n \to \infty} P (S | G) \notin {0, 1}$ .^[13] The accuracy and threshold likewise determine the limiting probability that the paper will erroneously not be written.
Hence ${lim}_{n \to \infty} P (mistake) \notin {0, 1}$ .^[14] The chance of the collective decision being mistaken in the limit is a mixture of (2) and (3).

There we have it—the Rational Unilateralist's Absolution: even in the limit, the risky paper might not be written. But the probability of such a mistake is very sensitive to parameters other than $n$ , and can be close to either zero or one. So even Bostrom et al might've been too generous in concluding that "the unilateralist curse is lifted" when players are appropriately rational (2016, p. 9).

In the graphs below, notice (i) that the risks can rise or fall with $n$ , (ii) how much they differ based on the other parameters, and (iii) how fast they converge to their limits.^[15]

Typically, the higher the threshold for publication, the lower the chance of writing a dangerous paper, even in the limit. This is good news, but it trades off against the chance of mistakenly staying silent. So, for all intents and purposes, rational unilateralists aren't really free of error. Granted, they don't converge to always publishing harmful papers, unlike irrational ones. But they're still rather lousy.

Thankfully, there is a way to lower the probability of a mistake far faster: just collaborate. The setup of this model satisfies the requirements for Condorcet's Jury Theorem, so if the agents decided by majority vote, they'd quickly converge to the correct decision.^[16] Here's just how much earlier majority vote could give the agents confidence that they won't mistakenly publish the paper:

Conclusion

Unilateral action is not so cursed when agents are rational. This detail is often omitted. A rational agent conditions the value of their options on the possible worlds in which their decision matters. And this liberates the group from the prospect of inevitable mistakes.

I simply find this interesting in its own right. Extracting practical upshots from this is something I’ll—and I’ve always wanted to do this—leave as an exercise to the reader. As a suggestion for somewhere to begin, I'll note that there is some empirical work testing how rational humans are in these settings. For instance, Guarnaschelli et al (2000) find that individual people's behaviour in juries does track the unobvious strategic considerations of unanimity, though the aggregate group decision is less well predicted by the theory. In contrast, Hendricks et al (2003) evaluate the related Winner's Curse in federal auctions and find impressively rational behaviour from firms.

^{^}
Bostrom et al (2016) compare the Unilateralist’s Curse to the Winner’s Curse from auction theory. There, too, the agents are assumed to be irrational.
^{^}
The framework is somewhat different from that in Bostrom et al (2016) but more in line with Feddersen and Pesendorfer (1998). This doesn’t meaningfully affect the conclusions.
^{^}
Bostrom et al (2016) mention the assumption of identical priors is 'problematic' but as should become clear in the derivation, denying this does not quite salvage their result (especially if the accuracy of the signal is high).
^{^}
The former is a normalisation but the latter is a simplifying assumption. I relax it in the next section.
^{^}
This is because the game is simultaneous. But note that timing in game theory is a property of the information environment, not of when agents act.
^{^}
Denote by $D$ the ‘decisive’ case where nobody else has chosen to write the paper. Then $i$ 's expected payoffs for writing and staying silent, respectively, are
$E [u_{i} (w) | m_{i}] = P (D | m_{i}) P (G | D, m_{i}) + [1 - P (D | m_{i})] P (G | \neg D, m_{i})$
$E [u_{i} (s) | m_{i}] = P (D | m_{i}) P (W | D, m_{i}) + [1 - P (D | m_{i})] P (G | \neg D, m_{i})$
And $i$ prefers writing to silence whenever $E [u_{i} (w) | m_{i}] > E [u_{i} (s) | m_{i}]$ so when
$P (D | m_{i}) P (G | D, m_{i}) + [1 - P (D | m_{i})] P (G | \neg D, m_{i})$
$> P (D | m_{i}) P (W | D, m_{i}) + [1 - P (D | m_{i})] P (G | \neg D, m_{i})$
$⟹ P (D | m_{i}) P (G | D, m_{i}) > P (D | m_{i}) P (W | D, m_{i})$
$⟹ P (G | D, m_{i}) > P (W | D, m_{i})$ , assuming $P (D | m_{i}) \neq 0$ .
So, $i$ ’s decision to write the paper is only a function of $i$ ’s beliefs conditional on being decisive.
^{^}
It’s assumed that $p \in (\frac{1}{2}, 1)$ and $n \geq 3$ .
^{^}
To compare directly to the derivations above, set $q = \frac{1}{2}$ .
^{^}
See the Fundamental Theorem of Mixed Strategy Nash Equilibrium.
^{^}
The astute reader may have noticed that $p_{g}$ can equal $0$ or $1$ . Does this contradict the proof above? Not quite. We still guarantee $p_{g} \in (0, 1)$ when $q = \frac{1}{2}$ , for example, which is implicit in the setup studied above. But in this more general case, we can indeed find equilibria in pure strategies.
We guarantee $p_{g} > 0$ when $p > q$ , and the converse sometimes holds. To motivate this, note that when agents play a pure strategy of never writing the paper, $p_{g} = 0$ , decisiveness is uninformative, so agents can only use their private signal to decide whether to write. And so if their confidence is above the threshold, $p > q$ , they’ll write.
We can get $p_{g} = 1$ when the threshold, accuracy, and number of agents is sufficiently low. To see why, consider the case of three agents with $q = 0.1$ and $p = 0.6$ . The posterior on publication being good then becomes $^{3} C_{1} (1 - p)^{2} p \approx 0.23$ , which is above the threshold. If you think beneficial publication is sufficiently better than faulty publication is bad, you might write the paper even if you think publication is likely to cause harm.
^{^}
${lim}_{n \to \infty} (p_{g}^{'} (n)) = \frac{{[\frac{(1 - q) p}{q (1 - p)}]}^{lim (\frac{1}{n - 1})} - 1}{p ⎛ ⎜ ⎜ ⎜ ⎝ 1 + {[\frac{(1 - q) p}{q (1 - p)}]}^{lim (\frac{1}{n - 1})} ⎞ ⎟ ⎟ ⎟ ⎠ - 1} = \frac{{[\frac{(1 - q) p}{q (1 - p)}]}^{0} - 1}{p (1 + {[\frac{(1 - q) p}{q (1 - p)}]}^{0}) - 1} = \frac{1 - 1}{p (1 + 1) - 1} = 0.$
^{^}
$P (W | B) = 1 - P (s | B)^{n}$
$= 1 - [P (s | m_{i} = b) P (m_{i} = b | B) + P (s | m_{i} = g) P (m_{i} = g | B)]^{n}$
$= 1 - [p + (1 - p) (1 - p_{g} (n))]^{n}$ .
So ${lim}_{n \to \infty} P (W | B) = 1 - (\frac{q (1 - p)}{(1 - q) p})^{\frac{1 - p}{2 p - 1}}$ (see Feddersen and Pesendorfer 1998).
^{^}
$P (S | G) = P (s | G)^{n}$
$= [P (s | m_{i} = b) P (m_{i} = b | G) + P (s | m_{i} = g) P (m_{i} = g | G)]^{n}$
$= [1 - p + p (1 - p_{g} (n))]^{n}$
$= [1 - p p_{g} (n)]^{n}$ .
So ${lim}_{n \to \infty} P (S | G) = (\frac{q (1 - p)}{(1 - q) p})^{\frac{p}{2 p - 1}}$ (see Feddersen and Pesendorfer 1998).
^{^}
$P (mistake) = P (W | B) P (B) + P (S | G) P (G)$ .
^{^}
Remember only integers of $n$ are meaningful.
^{^}
The probability of majority rule making a mistake in this context is $P (W | D)_{Maj} = 1 - \sum_{k = \frac{n + 1}{2}}^{n}^{n} C_{k} p^{k} (1 - p)^{n - k}$ for odd $n$ .