Consider a simple coordination game. In this game, two players (player 1 and player 2) each simultaneously choose an action, X or Y. If they both choose X, they both get 1 utility. If they both choose Y, they both get utility for some known to both players. If they choose different actions, they both get 0 utility. Which action should they each choose? Assume they get to communicate before knowing , but not after knowing .
An optimal policy pair is for each to pick X if , and Y otherwise. Unfortunately, this policy pair can break down in the presence of even a small amount of noise. Assume neither player observes , but instead each receives an independent observation (player 1 sees , player 2 sees ), each of which is drawn uniformly from the range for some small . If both players follow the policy of choosing X if and only if their observation of is less than 1, then if is very close to 1, there is a significant likelihood that one player will receive an observation less than 1, while the other will receive one greater than 1. Thus, they have a significant chance of choosing different actions and receiving 0 utility. This issue is essentially the same as the one faced by Buridan's ass: both options are equally good, so there is no way to effectively decide between them.
In fact, as I will prove:
Claim 1: No pair of policies in which the players select their actions independently given their observations can simultaneously guarantee an expected utility greater than for all .
But things change when a shared source of randomness is allowed. Then, as I will also prove:
Claim 2: Suppose that, after observing their observation of , each player also gets to observe a random number . Then there is a pair of policies that achieves expected utility at least regardless of .
This solution method extends to arbitrary cooperative normal-form games where players observe a perturbed version of the true utility function. This is shown in the appendix.
Why is this important? I expect progress in decision theory to come from studying very simple decision problems like this one. If it is possible to show that any solution to some simple problem has certain properties, then this usefully constrains the design space of possible decision theories. In this case, the result suggests that something like shared randomness may be required for achieving provable near-optimality even in cooperative settings.
Claim 1: impossibility of solving the game using independent randomness
Fix . Let be Lesbegue measurable functions representing the two players' policies, which map to the player's probability of choosing action Y.
Define to be the function mapping to the resulting distribution.
Define . This is defined so that equals player 's overall probability of choosing action Y.
For , note that the total variation distance between and is at most . Thus, since is bounded between 0 and 1, is -Lipschitz and therefore continuous.
I will now show that, for some , the players achieve expected utility at most by case analysis on :
- If then at most 0.5 expected utility is achieved when (since at most 1 utility is achieved when player 1 chooses action X, and no utility is achieved when player 1 chooses action Y).
- If , then at most 1.5 expected utility is achieved when (since at most 1 utility is achieved when player 1 chooses action X, and at most 2 utility is achieved when player 1 chooses action Y).
- If neither of the two above cases hold, then by continuity of and the intermediate value theorem, there exists some with . When , since the two players choose actions independently, there is a 0.5 probability that they select the same action, ensuring that they achieve at most expected utility.
Thus, claim 1 is proven. This proof method bears resemblance to the problem faced by Buridan's ass: in the third case, some value is found so that the "policy" is equally compelled by both options, choosing between them with a 50/50 coin flip in a way that is disastrous for coordination.
Claim 2: solving the game using shared randomness
Define . Consider a pair of policies in which player chooses Y if , and otherwise chooses X, where . Intuitively, each of these policies increases its chance of taking action Y as its observation of increases in a smooth fashion, and they correlate their randomness so that they are likely to choose the same action. Now note a couple properties of this pair of policies:
1. For , it is guaranteed that , so both players always take action Y.
2. Since is -Lipschitz, and , the probability of the players taking different actions is at most .
I will now show that the players' expected utility is at least by case analysis:
- Suppose . Then by property 1, both players are guaranteed to take action Y, so they achieve expected utility .
- Suppose . By property 2, the players choose the same action with probability at least , and taking the same action yields a utility of at least 1, so they achieve expected utility at least . Due to the bound on , this is at least .
Thus, the claim is proven. By setting sufficiently low, this pair of policies yields an expected utility arbitrarily close to regardless of .
Conclusion and directions for further research
Together, these claims show that there is a simple set of noisy coordination games (namely, the set of games described in the beginning of this post for all possible values) that is impossible to solve with only independent randomness, but which can be solved using shared randomness. Some notes on this:
- The proof of claim 1 only used the fact that is continuous. So even without noise, if the players' policies are a continuous function of the utility , the same problem occurs.
- A pair of Bayesians playing this coordination game who have a prior over have no need for randomization, independent or joint. The goal of achieving a high expected utility regardless of most clearly makes sense if is selected adversarially (to maximize regret). It also makes sense if the environment is hard to model in a way that makes selection of a policy with a low rate of "ties" difficult. The comparison between the shared randomness solution and the Bayesian solution is similar to the comparison between randomized Quicksort and a "Bayesian" variant of Quicksort that selects pivots based on some expected distribution of inputs. While the Bayesian solution works better than the randomized solution when the prior over the input distribution is accurate, the randomized solution is simpler, is amenable to proofs, and works well even under adversarial conditions.
Where to go from here? Roughly, my overall "plan" for formal decision theory consists of 3 steps:
1. Solve Cartesian cooperative decision theory problems where players reason about each other using something like reflective oracles.
2. Extend the solution in 1 to Cartesian multiplayer game theory problems where players have different utility functions and reason about each other using something like reflective oracles.
3. Extend the solution in 2 to a logically uncertain naturalized setting.
Step 1 has still not been solved. Previous writing on this includes this post which studies a setting with a single memoryless player (which is similar to a set of players who have the same utility function). The post shows (redundantly with the paper introducing the absent-minded driver problem as I later found out) that, if the player's policy is globally optimal (i.e. it achieves the highest possible expected utility), then all actions that might be taken by that policy are CDT-optimal, assuming SIA probabilities. This second condition is a local optimality condition, so the post shows that global optimality implies local optimality.
It would be highly desirable to find a similar but different local optimality property that implies a global optimality property. That would essentially be a way of deriving collective self-interest from individual self-interest when individuals have the same utility function and common priors.
As this current post shows, the globally optimal policy is discontinuous as a function of the utilities involved, and no continuous approximation always yields a near-optimal expected utility. I expect this to hinder attempts to derive global optimality from local optimality, as it implies there is a valley of bad policies between decent policies and good ones.
Introducing shared randomness here may help by preserving continuity. So a natural next step is to find useful local optimality properties in a cooperative setting where players have shared randomness.
Appendix: extension to arbitrary normal-form cooperative games
The solution described in the section on claim 2 can be extended to arbitrary normal-form cooperative games where the players receive only noisy observations of the payoffs. Reading this appendix (which takes up the rest of this post) is unnecessary for understanding the main point of this post.
Let be the number of players, be player 's set of actions, and be the shared unknown utility function over strategy profiles, whose minimum value is and whose maximum value is .
Fix . Let be a "perturbed" version of that player observes; it must satisfy the property that for any strategy profile , .
To define the policies, we will first number the strategy profiles arbitrarily where . Let be some number to be determined later. For , define
This defines, given any , a probability distribution over strategy profiles, since . Specifically, the distribution is a softmax. This distribution is more likely to select strategy profiles that considers better, but has some chance of selecting every strategy profile.
Players will sample from this distribution using a source of shared randomness, . Define
Now note that , i.e. is distributed according to . Define policies . That is, player will play their appropriate action for strategy profile number .
Roughly, we will now show 2 properties that are sufficient to establish that these policies are near-optimal:
- With high probability, the players play according to , i.e. for all , .
- The strategy profile is near-optimal in expectation, i.e. is close to .
Players play according to with high probability
First, we will show that, for all and , is close to .
For all , since , we have
Therefore, for all ,
By identical logic,
Now we will bound .
For it to be the case that , it must be the case that for some ,
Due to the bound on , the measure of this hull is at most . Furthermore, there are hulls of this form (one for each value), so the total measure of these hulls is at most .
So, player chooses action with probability at least . Thus, by the union bound, the players jointly choose the actions with probability at least .
is near-optimal in expectation
First we will prove a lemma.
Softmax lemma: For any and vector of length ,
Let be the maximum of . Now:
For any , we have . Therefore for any , . By exponentiating both sides, .
Applying this to the term from the previous inequality:
At this point we have
At this point the implication is straightforward. Since is distributed according to , we have
Proving the result from these facts
At this point we have:
- Players play according to with probability at least .
The first fact lets us quantify the expected difference between and . Since these random variables are equal with probability at least , and their difference is bounded by , their expected difference is at most .
Combining this with the second inequality:
To minimize , set
Now note that
Due to the fact that and ,
Because for any , , we have
By combining inequalities and equalities so far:
The expected suboptimality goes to 0 as approaches 0.
This doesn't (I think) really have much to do with randomness as such. The relevant thing about R is that it's shared information that a hypothetical adversary doesn't get to see.
If uy isn't chosen adversarially, then our players don't care about pessimizing over uy but about something like an average over uy, and then R isn't needed. Or, if they are ultra-cautious people who universally care about worst cases, then they don't care about expectation w.r.t. R but about the worst case as R varies, and then R doesn't help. R only helps them when uy is chosen adversarially and R isn't; or when they care about pessimizing w.r.t uy but not w.r.t. R.
So the real conclusion here is not "sometimes shared randomness helps", it's "if some people are trying to coordinate in the presence of an adversary, anything they share and the adversary has no access to helps".
Sometimes you want to prove a theorem like "The algorithm works well." You generally need randomness if you want to find algorithms that work without strong assumptions on the environment, whether or not there is really an adversary (who knows what kinds of correlations exist in the environment, whether or not you call them an "adversary").
A bayesian might not like this, because they'd prefer prove theorems like "The algorithm works well on average for a random environment drawn from the prior the agents use," for which randomness is never useful.
But specifying the true prior is generally hideously intractable. So a slightly more wise Bayesian might want to prove statements like "The algorithm well on average for a random environment drawn from the real prior" where the "real prior" is some object that we can talk about but have no explicit access to. And now the wiser Bayesian is back to needing randomness.
It seems like a bayesian can conclude that randomness is useful, if their prior puts significant weight on "the environment happens to contain something that iterates over my decision algorithm and returns its worst-case input, or something that's equivalent to or approximates this" (which they should, especially after updating on their own existence). I guess right now we don't know how to handle this in a naturalistic way (e.g., let both intentional and accidental adversaries fall out of some simplicity prior) and so are forced to explicitly assume the existence of adversaries (as in game theory and this post).
This seems basically right. As discussed in the conclusion, there are reasons to care about worst-case performance other than literal adversaries.
Sure, but non-adversarial cases (really, any cases where u is determined independently of strategies chosen) can just choose R as a fixed part of the strategy, rather than a random shared component determined later.
That's right, but getting the worst-case guarantee requires this initial choice to be random.
Nope. Random choice gives a specific value for R each game. The outcome for that iteration is IDENTICAL to the outcome if that R was chosen intentionally. Randomness only has game value as a mechanism to keep information from an adversarial actor.
To be clear, by "worst-case guarantee" I mean "the expected utility is guaranteed to be pretty good regardless of uy", which is unattainable without shared randomness (claim 1).
I think you are either misunderstanding or disagreeing with a lot of the terminology on randomized algorithms and worst-case guarantees that are commonly used in CS and statistics. This article is a decent introduction to this topic.
I'm missing something (and I haven't digested the math, so maybe it's obvious but just missing from the narrative description). Is epsilon the same for both players, in that they see the same V, it just may not exactly match u? or is it different for each player, meaning for the same u, they have different V? From your analysis (risk of 0), it sounds like the latter.
In that case, I don't see how additional shared knowledge helps coordinate them, nor why it needs to be random rather than just a fixed value they agree on in advance. And certainly not why it matters if the additional random shared value is generated before or after the game starts.
If they don't have this additional source of shared randomness, can they just decide in their pre-game discussion to use R=0.5? Why or why not?
ϵ is the same for both players but V1 and V2 (the players' observations of uy ) are different, both sampled independently uniformly from [uy−ϵ,uy+ϵ] .
If they decide on R=0.5 then there exists some uy value for which they get a bad expected utility (see Claim 1).
Sure, but that goes for a randomly-chosen R too. For every possible R, there is a u value for which they get bad outcomes. It doesn't get better by randomly choosing R.
The assumption is that R is chosen after uy . So for every uy the pair of policies gets a good expected utility. See the point on Bayesian algorithms in the conclusion for more on why "get a high expected utility regardless of uy " might be a desirable goal.
How can it possibly matter whether R is chosen before or after uy? R is completely independent of u, right? It's not a covert communication mechanism about the players' observations, it's a random value.
If uy is chosen after R then it might be chosen to depend on R in such a way that the algorithm gets bad performance, e.g. using the method in the proof of Claim 1.
Based on other comments, I realize I'm making an assumption for something you haven't specified. How is uy chosen? If it's random and independent, then my assertion holds, if it's selected by an adversary who knows the players' full strategies somehow, then R is just a way of keeping a secret from the adversary - sequence doesn't matter, but knowledge does.
Claim 1 says there exists some uy value for which the algorithm gets high regret, so we might as well assume it's chosen to maximize regret.
Claim 2 says the algorithm has low regret regrardless of uy , so we might as well assume it's chosen to maximize regret.
uy and R are independently chosen from well-defined distributions. Regardless of sequence, neither knows the other and CANNOT be chosen based on the other. I'll see if I can find time tonight to figure out whether I'm saying your claim 1 is wrong (it dropped epsilon too soon from the floor value, but I'm not sure if it's more fundamentally problematic than that) or that your claim 2 is misleading.
My current expectation is that I'll find that your claim 2 results are available in situation 1, by using your given function with a pre-agreed value rather than a random one.
The theorems are of the form "For all uy, you get good outcomes" or "There exists a uy that causes bad outcomes".
When you want to prove statements of this form, uy is chosen adversarially, so it matters whether it is chosen before or after R.
What distribution is uy chosen from? That's not specified anywhere in the post.
True, they will fail to cooperate for some R, but the values of such R have a low probability. (But yeah, it's also required that uy and R are chosen independently—otherwise an adversary could just choose either so that it results in the players choosing different actions.)
The smoothness comes in from marginalising a random R. The coordination comes from making R and ε common knowledge, so they cooperate using the correlation in their observations—an interesting phenomenon.
(How can I write LaTeX in the comments?)
If you assume a fixed probability distribution over possible $u_y$ that both players know when coordinating, then they can set up the rules they choose to make sure that they probably win. The extra random information is only useful because of the implicit "for all $u_y$". If some malicious person had overheard their strategy, and was allowed to choose $u_y$, but didn't have access to the random number source, then the random numbers are useful.
I expected that Lamport paper would be mentioned, as it describes a known catastrophic mode for autonomous systems, connected with Buridan ass problem and infinite recursion about predicting future time of the problem solving. I think that this problem is underexplored for AI Safety, despite previous attempt to present it on LessWrong.
Is this a reinvention of correlated equilibrium?
This is not a reinvention of correlated equilibrium, although that is related.
Looks like I was wrong again and shame on me. Correlated equilibrium can't be useful in a purely cooperative game, it's only useful if we're trying to minimax over many cooperative games. You even spelled it out in the post, but my eyes skipped over it. Sorry again.
Yeah, sorry, I was hasty. Looks like you're making a more interesting point: correlated equilibrium can be useful even in purely cooperative games. I wonder what would be the simplest example game...