I recently realized that the formalism of incomplete models provides a rather natural solution to all decision theory problems involving "Omega" (something that predicts the agent's decisions). An incomplete hypothesis may be thought of a zero-sum game between the agent and an imaginary opponent (we will call the opponent "Murphy" as in Murphy's law). If we assume that the agent cannot randomize against Omega, we need to use the deterministic version of the formalism. That is, an agent that learns an incomplete hypothesis converges to the corresponding maximin value in pure strategies. (The stochastic version can be regarded as a special case of the deterministic version where the agent has access to an external random number generator that is hidden from the rest of the environment according to the hypothesis.) To every decision problem, we can now correspond an incomplete hypothesis as follows. Every time Omega makes a prediction about the agent's future action in some counterfactual, we have Murphy make a guess instead. This guess cannot be directly observed by the agent. If the relevant counterfactual is realized, then the agent's action renders the guess false or true. If the guess is false, the agent receives infinite (or, sufficiently large) reward. If the guess is true, everything proceeds as usual. The maximin value then corresponds to the scenario where the guess is true and the agent behaves as if its action controls the guess. (Which is exactly what FDT and its variants try to achieve.)

For example, consider (repeated) counterfactual mugging. The incomplete hypothesis is a partially observable stochastic game (between the agent and Murphy), with the following states:

• $s_0$: initial state. Murphy has two actions: $g_{+}$ (guess the agent will pay), transitioning to $s_{1+}$ and $g_{-}$ (guess the agent won't pay) transitioning to $s_{1-}$. (Reward = $0$)
• $s_{1+}$: Murphy guessed the agent will pay. Transitions to $s_{2a+}$ or $s_{2b+}$ with probability $\frac{1}{2}$ to each (the coin flip). (Reward = $0$)
• $s_{1-}$: Murphy guessed the agent won't pay. Transitions to $s_{2a-}$ or $s_{2b-}$ with probability $\frac{1}{2}$ to each (the coin flip). (Reward = $0$)
• $s_{2a+}$: Agent receives the prize. Transitions to $s_{3u}$. (Reward = $+1$)
• $s_{2b+}$: Agent is asked for payment. Agent has two actions: $p_{+}$ (pay) transitioning to $s_{3r+}$ and $p_{-}$ (don't pay) transitioning to $s_{3w-}$. (Reward = $0$)
• $s_{2a-}$: Agent receives nothing. Transitions to $s_{3u}$. (Reward = $0$)
• $s_{2b-}$: Agent is asked for payment. Agent has two actions: $p_{+}$ (pay) transitioning to $s_{3w+}$ and $p_{-}$ (don't pay) transitioning to $s_{3r-}$. (Reward = $0$)
• $s_{3u}$: Murphy's guess remained untested. Transitions to $s_0$. (Reward = $0$)
• $s_{3r+}$: Murphy's guess was right, agent paid. Transitions to $s_0$. (Reward = $-0.1$)
• $s_{3r-}$: Murphy's guess was right, agent didn't pay. Transitions to $s_0$. (Reward = $0$)
• $s_{3w+}$: Murphy's guess was wrong, agent paid. Transitions to $s_0$. (Reward = $+1.9$)
• $s_{3w-}$: Murphy's guess was wrong, agent didn't pay. Transitions to $s_0$. (Reward = $+2$)

The only percepts the agent receives are (i) the reward and (ii) whether it is asked for payment or not. The agent's maximin policy is paying, since it guarantees an expected reward of $\frac{1}{2}\cdot 1+\frac{1}{2}\cdot (-0.1)=0.45$ per round.

We can generalize this to an imperfect predictor (a predictor that sometimes makes mistakes), by using the same construction but adding noise to Murphy's guess for purposes other than the guess's correctness. Apparently, We can also generalize to the variant where the agent can randomize against Omega and Omega decides based on its predictions of the probabilities. This, however, is more complicated. In this variant there is no binary notion of "right" and "wrong" guess. Instead, we need to apply some statistical test to the guesses and compare it against a threshold. We can then consider a family of hypotheses with different thresholds, such that (i) with probability $1$, for all but some finite number of thresholds, accurate guesses would never be judged wrong by the test (ii) with probability $1$, consistently inaccurate guesses will be judged wrong by the test, with any threshold.

The same construction applies to logical counterfactual mugging, because the agent cannot distinguish between random and pseudorandom (by definition of pseudorandom). In TRL there would also be some family of programs the agent could execute s.t., according the hypothesis, their outputs are determined by the same "coin flips" as the offer to pay. However, this doesn't change the optimal strategy: the "logical time of precommitment" is determined by the computing power of the "core" RL agent, without the computer "envelope".