1 Introduction

This post will introduce our new paper "Pitfalls of Learning a Reward Function Online", now online for IJCAI2020.

It shows some of the difficulties with things we might think of as "preference learning processes", and the useful conditions we could impose to get around these.

The tl;dr summary is:

1. Things that seem like preference learning processes - including "have a prior, update it based on evidence" - have problems that allow the AI to manipulate the process.
2. Some things that don't seem like learning processes at all, actually are.
3. Part of the problem is that learning preferences is not well-grounded - we have to specify a learning process that allows the AI to connect facts about the world with facts about preferences.
4. There are many ways of specifying these, and most have problems.
5. Forget about "capturing the correct variable in the outside world"; it's tricky to design a learning process that "captures ANY variables in the outside world".
6. Thus we'll start by abstractly defining what a "preference learning process" is a very general way, rather than worrying about what we're learning: "how to learn" precedes "what to learn".
7. Then we'll add two useful conditions for such processes: unriggability, which implies the process respects conservation of expected evidence, and uninfluenceability, which implies the process derives from learning background variables in the environment.
8. We've shown that the syntactic/algebraic condition of unriggability is (almost) equivalent to the semantic condition of uninfluenceability.
9. Finally, we've shown that if the learning process is neither unriggable nor uninfluenceable, then the AI can manipulate the learning process, and there are situations where the AI's optimal policy is to sacrifice, with certainty, reward for every possible reward function.

1.1 Blast from the past: misleadingly named tokens

Good Old-Fashioned AI (sometimes called symbolic AI) did not work out. To define something, it wasn't enough to just name a token, and then set it up in relation to a few other named tokens, according to our own intuition about how these tokens related.

Saying "happiness is a state of mind", or "light is a wave", isn't nearly enough to define "happiness", "state of mind", "light", or "wave".

Similarly, designating something as "learning", and giving it some properties that we'd expect learning to have, isn't enough to make it into learning. And, conversely, sometime things that don't look like learning, behave exactly like they are.

2 What is learning anyway?

2.1 A simple prior-update process?

A coin is flipped and left on a rock somewhere. You may access the coin in one hour's time, for a few minutes. What's your probability that in two hours, the coin will be showing heads (event $H_2$) or tails (event $T_2$)?

Well, a reasonable prior would be to put a probability of $0.5$ on both possibilities, and then update based on your last observation in an hour (call this $H_1$ or $T_1$). Obviously^[1] $P(H_2\mid H_1)=P(T_2\mid T_1)=1$. So we have a prior and a (pretty trivial) update process. Is this basically learning?

Well, one thing I've implied but not stated: when you "access" the coin, you can pick it up and flip it before putting it back.

Nothing about this changes any of the probabilities I've mentioned. If you flip it to heads, then your last observation will indeed be heads.

This looks like pure manipulation of outcomes. But it also looks, formally, like a prior and updating process. So what is it?

Note that this process, whatever it is, violates conservation of expected evidence because

$$1=P(H_2\mid \text{flip to heads})\ne P(H_2\mid \text{flip to tails})=0.$$

But part of the reason that this violates the conservation law, is because events like "$\text{flip to heads}$" do not have well-defined probabilities for the agent who might (or might not) do the flipping. But that's the case for all traditional decision theory setups. So how can we still reason in these setups?

2.2 "Asking" the guards

To expand, suppose an AI meets those traditional guards, one of whom lies while the other tells the truth:

There's an AI, designed to give humans what they prefer. And it wants to figure out whether humans prefer cake or death. It has three options:

1. Immediately ask one of the guards at random, and take their answer as the truth.
2. Carefully figure out which guard tells the truth, and, when it is $99\%$ sure of this, ask the truth-teller, and take their answer as the truth.
3. Ignore the guards entirely, write down on "cake" or "death" on the ground, depending on which one is easiest for it to maximise. Take what's written down as the truth.

Now, 3. is clearly not what we'd want: it's another case of "flip the coin". But which of the other two processes are unambiguously well-behaved preference learning processes? It turns out 1. is a proper learning process, but 2. is not. How so?

2.3 Suggestively described "learning" processes

Let's deal with 2. first.

Why is 2. not a well-behaved preference learning process? Well, suppose that the AI has established, with probability $98\%$, that the red guard is the truth-teller. And suppose it's also established they are likely to say "cake", and, finally, suppose it would be much easier to maximiser "death" than "cake".

In that case, what's the utility-maximising thing to do? It wants to avoid going to $99\%$, because then it would be committed to cake-maximising ("take their answer as the truth").

So it could leave, and never enquire further, so it would never have to update that way. If it was powerful enough, it could instead try and take control of the truth-telling guard, and force them to answer "death" instead. It can take its time doing so, because it doesn't have to update anything until its credence goes to $99\%$.

So even though 2. seemed a very plausible description of a truth-learning process, it turns out not to be. Similarly, if you mixed Bayesian updating with function approximation - then this is also not proper learning, and may be exploitable.

2.4 Learning compound truths

But why is the process 1. a proper preference learning process?

Well, assume the asked guard said "cake". Now the AI knows that one of the following is true:

• That guard is a truth-teller, and humans prefer cake.
• That guard is a liar, and humans prefer death.

It has entirely ruled out:

• That guard is a truth-teller, and humans prefer death.
• That guard is a liar, and humans prefer cake.

So the AI has learnt, cutting the space of possibilities in half. It might not have learnt exactly what we wanted it to, or in the most efficient manner, but it's unambiguously learning.

But what about the "take their answers as the truth" clause? Is the AI not "learning" the wrong thing?

Ah, but remember what I wrote about named tokens. Let's assume that $R_{\text{cake}}$ is the reward function that rewards the AI for giving humans cake (in the way that we'd expect). Similarly, $R_{\text{death}}$ rewards if for giving humans death.

We then have the answer of the first guard asked: "cake" or "death". Then we have the humans' "true" preferences, cake or death.

So its learning process is:

• "cake" $⟹$ $R_{\text{cake}}$.
• "death" $⟹$ $R_{\text{death}}$.

And this is a perfectly valid learning process.

2.5 Moral facts (not) in the universe

What humans actually wanted was for our true preferences to imply the AI's reward function, ie:

• cake $⟹$ $R_{\text{cake}}$.
• death $⟹$ $R_{\text{death}}$.

But, as I've shown, mere observable facts about the universe do not establish preferences. This is somewhat similar to Hume's "is-ought problem": we don't know preferences just from facts^[2].

So the AI doesn't have access to the "true" variables, cake or death. Neither do we, but, typically, we have a better intuitive idea of them than we can explicitly describe. Thus what we want is a process $\rho$, such that the AI can look at the history $h_t$ of its inputs and outputs, and deduce from that whether $R_{\text{cake}}$ or $R_{\text{death}}$ is the reward function to follow, in some suitably "nice" way.

We want it to be so that:

• cake $⟹$ "cake" $⟹$ $R_{\text{cake}}$.
• death $⟹$ "death" $⟹$ $R_{\text{death}}$.

And this, even though cake and death are ill-defined variables (and arguably don't exist).

• The process $\rho$ is necessary to bridge the is-ought gap between what is true in the world, and what preferences should be.

2.6 A bridge too general

But before we can even look at this issue, we have another problem. The bridge that $\rho$ builds, is too general. It can model the coin flipping example and processes 2. and 3. from the "ask the guard" scenario.

For process 2.: if $g_i$ is guard $i$, $g_T$ is the truth-telling guard, and $r_i$ is what guard $g_i$ revealed when asked, we have, for history $h_t$:

• $\left\{P\right(g_i=g_T|h_t)>0.99\}\wedge \{r_i=$ "cake"$\}⟹R_{\text{cake}}$.
• $\left\{P\right(g_i=g_T|h_t)>0.99\}\wedge \{r_i=$ "death"$\}⟹R_{\text{death}}$.

So this is also a possible $\rho$. Process 3. is also a possible $\rho$; let $o_{\text{cake}}$ mean observing "cake" written down on the ground (and conversely for $o_{\text{cake}}$), then:

• $o_{\text{cake}}\in h_t⟹R_{\text{cake}}$.
• $o_{\text{death}}\in h_t⟹R_{\text{death}}$.

So, before even talking about whether the AI has learnt from the right variables in the environment, we have to ask: has the AI "learnt" about any actual variables at all?

We need to check how the AI learns before thinking about what it's learning.

3 Formalism and learning

#3.1 Formalism

To get any further, we need to more formalism. So imagine that the AI has interacted with the world in a series of time steps. It will start by taking action $a_1$, and get observation $o_1$, take action $a_2$, get observation $o_2$, and so on. By turn $t$, it will have seen a history $h_t=a_1{o}_1\dots a_t{o}_t$.

We'll assume that after $n$ turns, the AI's interaction with the environment ceases^[3]; call $H_n$ the set of "complete" histories of length $n$. Let $R$ be the set of all possible reward functions (ie the possible preferences we'd want the AI to learn). Each $R\in R$ is a function^[4] from $H_n$ to $R$.

So, what is a learning process^[5] $\rho$? Well, this is supposed to give us a reward function, given the history the AI has observed. This need not be deterministic; so, if $\Delta \left(R\right)$ is the set of probability distributions over $R$,

• $\rho :H_n\to \Delta \left(R\right)$.

We'll see later why $\rho$ is defined for complete histories, not for all histories.

3.2 Policies, environments, and causal graphs

Are we done with the formalisms yet? Not quite. We need to know where the actions $a_t$ and the observations $o_n$ come from.

The actions are generated by the AI's policy $\pi$. This takes the history $h_t$ so far, and generates the next action $a_{t+1}$, possibly stochastically.

The observations come from the environment $\mu$. This takes $h_t{a}_{t+1}$, the history and action so far, and generates the next observation $o_{t+1}$, possibly stochastically. We don't assume any Markov condition, so this may be a function of the whole past history.

We can tie this all together in the following causal graph:

The rectangle there is 'plate notation': it basically means that, for every value of $j$ from $1$ to $n$, the graph inside the rectangle is true.

The $r$ is the AI's final reward, which is a function of the final history $h_n$ and the reward function $R$ (which itself is a function $h_n$ and $\rho$).

Ok, what flexibility do we have in assigning probability distributions to this graph? Almost all the arrows are natural: $h_j$ is a function of $h_{j-1}$, $a_j$, and $o_j$, by just... concatenating: $h_j=h_{j-1}{a}_j{o}_j$. Similarly, $\rho$ is a distribution on reward functions conditional on $h_n$, so the value of $R$ conditional on $\rho$ and $h_n$ is... $\rho \left(h_n\right)$.

There are three non-trivial nodes: $\pi$, $\mu$, and $\rho$. The $\pi$ is presumably set by the programmer, who would also want to design a good $\rho$. The $\mu$ is the environment, which we're not assuming is known by either the programmers or the AI. The AI will, however, have some prior over $\mu$.

4 Conditions for a good learning process

So, now that we have the formalism, how can we select a "good" (or "true" or "proper") learning process?

There are two plausible conditions: the syntactic condition of unriggability, and the semantic condition of uninfluenceability.

4.1 Unriggability

Unriggability comes from the idea of conservation of expected evidence. Basically, if $\rho$ defines a true learning process, we'd want it conserve expected evidence.

So, let's look at a version of the "look at the coin" situation. For example, let $R_H$ be a reward function defined by a coin being heads, and $R_T$ by it being tails. On the first turn, the coin has been tossed, but not seen. The AI can take an action $a_1=\text{look}$ to look, or $a_1=\neg \text{look}$ to not look.

This has conservation of expected evidence. If the AI choose to look:

• $P[P\left(R_H|h_1\right)=1|a_1=\text{look}]=0.5$.
• $P[P\left(R_T|h_1\right)=1|a_1=\text{look}]=0.5$.

This shows that, if the AI chooses to look, it will become certain about which reward function it must follow - but it's 50-50 about which reward function it will become certain about.

On the other hand, if the AI choose not to look:

• $P[P\left(R_H|h_1\right)=0.5|a_1=\neg \text{look}]=1$.
• $P[P\left(R_T|h_1\right)=0.5|a_1=\neg \text{look}]=1$.

This shows that, if the AI choose not to look, it will, with certainty, remain 50-50 uncertain about which reward function if must follow.

In expectation this is the same thing as if it chooses to look. So, the AI cannot change the expectation of its reward function^[6], but it might be able to change the variance. The future variance given $a_1=\text{look}$ is $0$ (it's absolutely certain about its reward function) while the future variance given $a_1=\neg \text{look}$ is $1/4$ (maximal uncertainty).

On the other hand, imagine there are two other actions, which involve the AI setting the coin to heads or tails, rather than looking at it what it fell on. If $a_1=\text{setH}$ or $a_1=\text{setT}$, then

• $P[P\left(R_H|h_1\right)=1|a_1=\text{setH}]=1$.
• $P[P\left(R_T|h_1\right)=1|a_1=\text{setT}]=1$.

This violates conservation of expected evidence, and, more importantly, it's the kind of behaviour we want to avoid: the AI setting its own reward. So one reasonable condition for $\rho$ would be:

• The reward-function learning process $\rho$ is unriggable if it respects conservation of expected evidence.

4.2 Uninfluenceability

Another thing that we'd want, is that preference learning should be like factual learning; ie it should depend on facts in the outside world.

In out setup "depend on facts in the outside world" can be taken to mean "depends on the environment $\mu$". This gives the following causal graph:

Here, the reward function $R$ is no longer (directly) a function of $h_n$, but instead is a function of $\mu$. The $\eta$ gives the conditional probability distribution over $R$, given $\mu$.

The connection between $\rho$ and $\eta$ is as follows: given a prior $p_{\mu }$ over $\mu$, the AI can use a history $h_n$ to update this prior to a posterior over environments. Then $\eta$ allows it to make this into a posterior over reward functions. Thus, given $p_{\mu }$ and $\eta$, the AI has a probability distribution over $R$ conditional on $h_n$; this is the $\rho$.

Thus define uninfluenceability:

• The reward function learning process $\rho$ is uninfluenceable if it derives (via the AI's prior) from a reward-function distribution $\eta$, conditional on the environment.

5 Results

Then our paper proves the following results:

• Every uninfluenceable preference learning process is unriggable.
• Every unriggable preference learning process is uninfluenceable, if the set of possible environments is large enough (though this may need to include "impossible" environments).
• If a preference learning process is unriggable, then it can be unambiguously defined over partial histories $h_t$, for $t<n$, rather than just for complete histories $h_n$.
• Every riggable preference learning process is manipulable in the following sense: there is always a relabelling of the reward functions, such that the AI's optimal policy is to sacrifice, with certainty, reward for every possible reward function.
• We can use a "counterfactual" approach to make a riggable learning process into an uninfluenceable learning process. This is akin to "what the truth-telling guard would have told you had you asked them immediately".

---

1. Let's ignore that, in reality, no probability is truly $1$ (or $0$). ↩︎

2. I don't want to get into the moral realism debate, but it seems that me and moral realists differ mainly in emphasis: I say "without making assumptions, we can't figure out preferences, so we need to find good assumptions", while they say "having made these good (obvious) assumptions, we can figure out preferences". ↩︎

3. There are ways of making this work with $n\to \infty$, but that extra complexity is not needed for this exposition. ↩︎

4. This is the most general reward function formats; if, for example, we had a Markovian reward function $S$ that just took the latest actions and observations as inputs, this defines an $R^S\in R$ such that $R^S\left(h_t\right)={\sum }_{i=1}^{t}S\left(a_i{o}_i\right)$. ↩︎

5. A terminological note here. We've decided to describe $\rho$ as a learning process, with "unriggable learning process" and "uninfluenceable learning process" being the terms if $\rho$ has additional properties. But the general $\rho$ includes things we might not feel are anything like "learning", like the AI writing down its own reward function.

   So it might make more sense to reserve "learning" for the unriggable processes, and call the general $\rho$ something else. But this is a judgement call, and people generally consider "ask your programmer" or "look at the coin" to be a learning processes, which are very much riggable. So I've decided to call the general $\rho$ a learning process. ↩︎

6. This "expectation" can be made fully rigorous, since reward functions form an affine space: you can take weighted averages of reward function, $(1-x)R_1+x{R}_2$. ↩︎