Summary: Reflective oracles correspond to Nash equilibria. A correlated version of reflective oracles exists and corresponds to correlated equilibria. The set of these objects is convex, which is useful.

(thanks to Sam for a lot of help with coming up with constructions)

## Motivation

Reflective oracles assign a probability to each query. The set of reflective oracles is not convex. For example, consider a machine (i.e. a machine that asks whether the probability that it returns 1 is at least 0.5). There are reflective oracles that assign probabilities 0, 0.5, and 1 to the query , and there aren't reflective oracles that assign other probabilities. So the set of possible reflective oracles isn't convex. This is for the same reason that the set of Nash equilibria isn't always convex.

Related to the non-convexity, there isn't in general a continuous way of mapping the numerical parameters of some Turing machines to a reflective oracle for those machines (or doing so as a Kakutani map). (Like there isn't a way of Kakutani-mapping the parameters of a game to an -approximate Nash equilibrium for that game).

This makes decision problems involving reflective oracles harder to analyze: the function mapping agents' policies to the resulting reflective oracle will be discontinuous.

So to analyze these decision problems, it might be useful to construct a convex set analogous to the non-convex set of reflective oracles. Luckily, the set of correlated equilibria in a game is convex. This post presents an analogue of them in reflective oracle land.

## Setup

An oracle (mapping from queries to answers) will be selected at random from some distribution. A query can ask about the distribution over oracles, conditional on the answer to that query.

## Definitions: machines, oracles, queries

Let be some finite set of Turing machines that ZFC-provably halt on every input. (We could in principle deal with both infinite sets of Turing machines and possibly-non-halting Turing machines, as ordinary reflective oracles do, but this complicates things somewhat).

An oracle maps each machine in to a natural number. The set of oracles is .

An oracle distribution is a probability distribution over oracles. The set of oracle distributions is .

A query to an oracle is a way of asking about an oracle distribution in an "argmax" fashion. A query is represented as a list of functions . Write ; in words, the query asks for some such that has maximum expectation on the oracle distribution. Note that:

- is non-empty for each
- for each , is convex

Let be the set of queries. Roughly, the set of allowed queries are those that partition the set of oracle distributions into a bunch of convex polygons; this allows for arbitrarily fine-precision queries about the distribution.

Let us interpret the output of each Turing machine (on the empty input) as a query. Some encoding scheme is necessary; I don't think it matters much. So assume we have a map . (The main reason for representing queries using Turing machines is to allow quining).

We will impose one additional restriction: for any machine , the query it outputs must not depend on the distribution over (i.e. the query is actually only a function of the joint distribution over each value other than itself). This is to avoid liar's paradoxes.

## Reflectivity

An oracle distribution is reflective if, for each , and each such that :

where is an oracle distribution formed by conditioning the oracle distribution on the event .

In words, an oracle distribution is reflective if each possible answer to each query is correct thing to say about the distribution over oracles conditional on the answer to that query.

**Theorem 1:** A reflective joint oracle exists.

**Proof:**

This oracle distribution will independently roll a die for each query to produce . Specifically, it is formed by a mapping . Let the function map to an oracle distribution through independent sampling.

Define , where indicates a Cartesian product, and is the set of probability distributions over the set (which is finite in this case). Note that is always non-empty and that is upper-hemicontinuous (by Berge's Maximum Theorem). So by Kakutani's Fixed Point Theorem, there is some mapping such that .

Consider the oracle distribution . For any , and , we have

where the second equality follows from (a) the fact that does not depend on the distribution over and (b) the fact that all values are independent under . So is reflective.

**Theorem 2:** The set of reflective oracle distributions is convex.

**Proof:**

Let be reflective oracle distributions. Let . Define . This proof will show that is reflective.

Let . Let . Let be such that . Consider 3 cases:

- . Then . But since , we have . So as desired.
- . This is analogous to case 1.
- . We have and . Note that is a convex combination of and (this is a basic property of mixture distributions: the conditioning of a mixture of components is a mixture of the conditionings of the components). The set is convex (from being a query); therefore , as desired.

## Correspondence with correlated equilibria

Reflective oracle distributions can be used to find correlated equilibria. Say we have a normal-form game with players, where each player selects an action , and receives utility . Define queries as follows:

where the machine computes a representation of the query (the recursion works through mutual quining). It's easy to show that reflective oracle distributions (for the set of machines ) correspond exactly with correlated equilibria in the game.

Thus, as ordinary reflective oracles naturally yield Nash equilibria when causal decision theorists using them play games with each other, so do reflective oracle distributions naturally yield correlated equilibria when causal decision theorists using them play games with each other.

-approximate correlated equilibria can be computed in polynomial time (see lectures 17+18 here). I conjecture that -approximate reflective oracle distributions over a finite set of queries can also be found in polynomial time, perhaps by reducing the problem of finding -approximate reflective oracle distributions to the problem of finding -approximate correlated equilibria.