In my previous post, I explained how a set of programs predicting each other using a probabilistic oracle and taking utility-maximizing actions will naturally play a Nash equilibrium. However, this seems to be suspiciously CDT-like. A program will, for example, betray in a Nash equilibrium when playing against itself.

Naturally, probabilistic oracle machines are a good way to formalize Newcomblike problems (since we have prediction), so we can explore alternative decision theories in this framework. Assume we have some probabilistic world program $W\left(O\right)$ that runs the world and calls the oracle (which is not considered part of the world), returning an outcome. There may be (probabilistic) agent programs embedded in this world program (as is common in formalizations of UDT). Different agents value outcomes differently: define $V_i\left(w\right)$ to be the utility player $i$ assigns to the outcome $W\left(O\right)=w$. I am using $V_i$ instead of $U_i$ to differentiate this utility function from the game theoretic utility function in the previous post.

Assume that player $i$'s output is $a_i\left(O\right)=O(E_i,1/2)$ for some machine $E_i$. This is a convenient assumption and it might be the case that this is enough, but I haven't actually proven this.

Here's an example of player 1 playing Prisoner's dilemma against player 2: $$W\left(O\right)=\text{let}x=O(E_1,1/2),y=O(E_2,1/2)\text{in}(x,y)$$ $$U_1(x,y)=x-10y$$ $$U_2(x,y)=y-10x$$ Note that $x$ and $y$ are the results of 2 independent calls to $E_1$. We assume that action 1 is defection and action 0 is cooperation. As a side note, perhaps I should be using monadic do-notation to show that these calls are independent, but this is simple enough that it doesn't seem necessary.

Here's an example of player 1 playing Newcomb's problem: $$W\left(O\right)=\text{let}x=O(E_1,1/2),y=O(E_1,1/2)\text{in}(x,y)$$ $$U_1(x,y)=1000x+1000000(1-y)$$ Here, action 1 is 2-boxing and action 0 is 1-boxing. $x$ is the player's actual action while $y$ is Omega's prediction of the player's action. Note the strong resemblance between Newcomb's problem and playing prisoner's dilemma against yourself.

How might agents in this environment act to maximize utility? In the previous post, we had a quantity we wanted to estimate, $E\left[\frac{U_i(a_{-i},1)-U_i(a_{-i},0)+1}{2}\right]$. We had $P(E_i=1)$ equal to this expectation, so $O(E_i,1/2)$ will return a utility-maximizing action for player $i$. The quantities $U_i(a_{-i},1)$ and $U_i(a_{-i},0)$ are causal counterfactuals, but maybe we want to replace them with other kinds of counterfactuals.

To do this, we could define masked oracles $\text{Mask}(s,p,q,O)$ for state $s$, probabilities $p$ and $q$, and oracle $O$ as follows:

$$P\left(\text{Mask}\right(s,p,q,O\left)\right(s,p)=1)=q$$ $$P\left(\text{Mask}\right(s,p,q,O\left)\right(s_2,p_2\left)1\right)=P\left(O\right(s_2,p_2)=1)\text{for}s\ne s_2\vee p\ne p_2$$

This is the same as $O$ except that, when given the query $(s,p)$, it returns 1 with probability $q$. Note that $\text{Mask}(s,p,q,O)$ might not be a reflective oracle even if $O$ is.

For a UDT-style decision theory, we would like to define the machine $E_i$ to compute a utility difference between oracular counterfactuals (similar to logical counterfactuals):

$$P\left(E_i\right(O)=1)=E\left[\frac{V_i\left(M\right(\text{Mask}(E_i,1/2,1,O)\left)\right)-V_i\left(M\right(\text{Mask}(E_i,1/2,0,0)\left)\right)+1}{2}\right]$$

In other words, we try masking the oracle to make our agent $O(E_i,1/2)$ return either always 1 or always 0, and then simulate the world twice with these different oracles and get the expected utility difference. It is not hard to confirm that this reasoner succeeds in cooperating on the prisoner's dilemma with itself and one-boxing in Newcomb's problem. However, since the masked oracle need not be reflective, it will not propagate any consequences of the change we made, and we get a few counterintuitive results, such as not cooperating in a prisoner's dilemma with a slightly different but logically equivalent program.

There's also an issue with Newcomb's problem when we add a step of indirection to Omega's prediction: $$W\left(O\right)=\text{let}x=O(E_1,1/2),y=O(a_1,1/2)\text{in}(x,y)$$ $$U_1(x,y)=1000x+1000000(1-y)$$

Since the counterfactual only modifies $O(E_1,1/2)$ and not $O(a_1,1/2)$, the player in this case will conclude that 2-boxing is correct.

So, maybe we want to get a modified version $O^{\prime }$ of $O$ so that $O^{\prime }(s,p)=q$, but $O^{\prime }$ is also reflective, and is probably mostly similar to $O$ when possible. However, we have no guarantee that $O^{\prime }$ actually exists. For example, if $M$ is a machine that always returns 1, we cannot have a reflective oracle in which $O^{\prime }(M,1/2)=0$.

Perhaps we can construct our $E_i$ machines so that such an oracle always exists. I'm not sure what approach makes sense here. This seems similar to the problem that playing chicken with the universe is meant to solve, but I don't know how well this translates to a context based on probabilistic oracles rather than logic.

If we succeeded in specifying more reasonable modified oracles, we might end up with a nice alternative formalism of UDT that can be computed using Nash equilibria.