My counterfactual Oracle design uses a utility function/reward function in order to train it to give the right predictions. Paul Christiano asked whether the whole utility function approach was necessary, and whether it could be replaced with a simple sequence predictor.

It turns out the answers are no, and yes. The counterfactual nature of the approach can be preserved, without needing a utility function.

Counterfactual sequence prediction

There is a boxed Oracle, $O$, which gets fed a series of observations $o_0,o_1,\dots o_{n-1}$, and must make a prediction $p_n$ about the next observation.

If the prediction was sent out into the world, then $O$ is attempting to make $p_n$ into a self-confirming prediction, with all the problems that this could entail.

However, we make $O$ into a counterfactual Oracle; on some occasions, the output $p_n$ is erased, and not seen by anyone. In that case, the Oracle will get a special message $e$ (for "erasure"), that will be added on after of $o_{n-1}$.

Then the job of the counterfactual Oracle is, given a history $h_{n-1}=o_0{p}_1{o}_1\dots p_{n-1}{o}_{n-1}$ (including past "$e$"s), to produce a prediction $p_n$ that is the best prediction for $o_n$ given the history $h_{n-1}e$. In terms of words, the Oracle is trying to best predict the next observation, given that erasure is assumed to happen (the definition of "best" depends on the details of the design).

If our setup is well designed and there is no information leakage, then any $o_n$ has a probability that is independent of the prediction $p_n$:

• $\forall o_n,h_{n-1},p_n,p_n^{\prime }:P(o_n\mid h_{n-1}e{p}_n)=P(o_n\mid h_{n-1}e{p}_n^{\prime })$.

Meaning of predictions

If we ran the oracle without the assumption that $e$ would be observed, we'd have the standard self-confirming predictor (that being the only correct predictions); with that assumption, it becomes a counterfactual predictor.

Though counterfactuals are conceptually tricky, it's actually easier to understand what a counterfactual predictor is doing than a general sequence predictor Oracle. We understand that a general sequence predictor is trying to find a self-confirming prediction - one where $p_n$ is the best estimate of $o_n$, given $h_{n-1}{p}_n$ - but unless we understand the structure of the world very well, we don't have a clue where that self-confirming prediction might be.

In contrast, the counterfactual prediction is easy to interpret: "had we not seen $p_n$, that is what $o_n$ would have been"; there are no loops in the definition.

Extent of manipulation via prediction

Note that the Oracle can be used to estimate the extent to which the prediction is manipulative, by contrasting its predictions for $o_n$ given the $h_{n-1}$, and given $h_{n-1}e$. This reveals the extent to which the prediction is potentially manipulative.

Non-explicit erasure

If the Oracle isn't given an explicit erasure message, but has a causal model of the world, we can still run the counterfactual Oracle by having it choose $p_n$ to be the best fit for the $o_n$, conditional on $h_n{p}_n$ and on the erasure happening.

Factoring out certain routes

Assume now that the AI is an active AI, that can take actions in the world, affecting the observation $o_n$. So we'll replace $p_i$ terminology with $a_i$; note that the predictions $p_i$ were already actions, just actions of a particular type in particular setup.

We want to have the AI only consider certain routes to influence $o_n$. This might be because there is a reward-function learning process involved, and we don't want the AI to influence the learning process. Or maybe there is a corrigibility button involved, and we don't want the AI to try and ensure it is pressed or not pressed.

In any case, label the banned routes of influence by the variables $b$:

Note that the Oracle case is just a special case of this, where $b$ corresponds to "humans reading the prediction", and there being no direct arrow from $a_n/p_n$ to $o_n$.

Factoring out

Assume that $\hat{a}$ is the default action. Then we want the AI to maximise some expected score $S$ of $o_n$ and $a_n$, but acting as if the values of $b$ were set by $\hat{a}$ rather than the value of $a_n$.

Let $\left\{{\beta }_i\right\}$ be the possible values of $b$. We require one more assumption: that if $b={\beta }_i$ is possible given $\hat{a}$, then it is also possible given any other action $a$ (technically, we only need for the conditional probabilities $P(o_n\mid b={\beta }_i,a_n=a)$ to make sense).

Then if the AI has sufficient data to estimate expressions like $P(b={\beta }_i\mid a)$, we act as if the AI was off-policy where $b$ is concerned. Namely, we get it to compute the action $a$ that maximises:

• ${\sum }_{{\beta }_i}P(b={\beta }_i\mid a_n=\hat{a},h_{n-1})E\left[S\right(o_n,a)\mid b={\beta }_i,a_n=a,h_{n-1}]$.

Note that if we had $a_n=a$ instead of $a_n=\hat{a}$ in that expression, then this would just be the classical $E\left[S\right(o_n,a)\mid a_n=a,h_{n-1}]$.

This construction is essentially the same as the counterfactually uninfluenceable agent design, with $b$ representing the weights of the different reward functions that the AI was to "learn".