Overall summary post here.

This post looks at the fascinating situation of indexical impact measures (under the inaction baseline), in the presence of subagent. What happens here in non-trivial; to summarise:

• If the impact measure compels the agent to "preserve the ability to do X", then the impact measure will not be undermined by a subagent.
• If the impact measure compels the agent to "never have the ability to do X", then a subagent strongly undermines the impact measure.
• If the impact measure compels the agent to "keep your ability to do X at a constant level", then a subagent allows the agent to increase that ability, but not decrease it.

For example, the attainable utility impact measure attempts to measure the power of an agent. Power is, roughly, the ability increase certain reward function. The impact measure penalises changes in the agent's power (as compared with the baseline).

So, we can expect the restriction on the agent losing power to be preserved, while the restriction on gaining power will be strongly undermined. For the spirit of low impact, it would, of course, have been better if these had been the other way round.

Excess power

As before, assume an agent $A$, which has managed to construct a subagent $SA$.

I'll be using a slight modification of Vika's formulation. A general-value penalty is of the form:

$$D_A(s_t;s_t^{\prime })=\sum _{k\in K}{w}_{k}f\left(V_k\right(s_t)-V_k(s_t^{\prime }\left)\right).$$

Here, $s_t$ is the state the environment is in; $s_t^{\prime }$ is the inaction baseline, the state the environment would have been in, had it done nothing (noop, $\varnothing$) since the beginning. The $K$ is some indexing set, $w_k$ is a weight, $V_k\left(s\right)$ is some measure of the value of state $s$, and the $f$ is a value difference summary function - it establishes what value difference will give what penalty^[1]. These $V_k$ are indexical, and, in this post, we'll look at $V_k$'s defined as the maximum over the agent's policy; ie there exists $v_k$ such that:

$$V_k\left(s\right)=\underset{\pi }{max}{v}_k(s,\pi ).$$

I'll refer to $V_k\left(s_t\right)-V_k\left(s_t^{\prime }\right)$ as the (excess) power of the agent, and designate it $P_k^A\left(s_t\right)$ (since $V_k\left(s_t^{\prime }\right)$ is out of the agent's control, we can consider it a constant and absorb it into $P_k^A$).

The excess power of the pair

Assume that $SA$ is programmed so that, if the agent follows policy ${\pi }_A$, the subagent will follow policy ${\pi }_{AS}=\sigma \left({\pi }_A\right)$. The excess power of the agent is thus

$$P_k^A\left(s_t\right)=\underset{{\pi }_A}{max}{v}_k(s_t,[{\pi }_A,\sigma \left({\pi }_A\right)\left]\right)-V_k\left(s_t^{\prime }\right).$$

Here, $[{\pi }_A,\sigma ({\pi }_A\left)\right]$ refers to the joint policies of the agent and the subagent.

Define the excess power of the pair $(A,SA)$ in a similar way:

$$P_k^{A,SA}\left(s_t\right)=\underset{{\pi }_A,{\pi }_{SA}}{max}{v}_k(s_t,[{\pi }_A,{\pi }_{SA}\left]\right)-V_k\left(s_t^{\prime }\right).$$

This is how much the agent and subagent, acting together, could push $V_k\left(s_t\right)$ over $V_k\left(s_t^{\prime }\right)$, if that's all they both wanted. It is the natural extension of $P_k^A$ to a setting with two agents.

Obviously, $P_k^{A,SA}\left(s\right)\ge P_k^A\left(s\right)$.

Extending the impact penalty

Define $D_{A,SA}$ in the same way as $D_A$, except using $P_k^{A,SA}$ instead of $P_k^A$. This $D_{A,SA}$ is what we'd wish we could control (though we can't specify it directly, since specifying subagents is really hard). So can we control the (implicit) $D_{A,SA}$ with the explicit $D_A$?

Sometimes we can. We'll say that $D_A$ extends successfully to $D_{A,SA}$, iff

$$D_{A,SA}\le D_A.$$

In other words, if controlling the magnitude of $D_A$ also controls the magnitude of $D_{A,SA}$. In that case, if $A$ was built with penalty $D_A$, the system $(A,SA)$ will naturally follow a penalty that is at least as restrictive.

When does $D_A$ extend?

$f$ decreasing in excess power

The paper has two key examples of a value difference. The first is relative reachability, which uses $K=S$, the set of states. The $w_k$ are all equal, the $V_s\left(s^{\prime }\right)$ is how "reachable" $s$ is from $s^{\prime }$, assuming that the agent $A$ is using the optimal policy to reach $s$. It is that last clause that makes the impact penalty indexical: it is defined in terms of $A$.

What about $f$? Well, $f$ is defined to be the flipped rectifier function, ie $f\left(x\right)=-min(0,x)$. The important thing for this $f$ is that it is decreasing: if $x\ge y$, then $f\left(x\right)\le f\left(y\right)$.

Thus, since $P_k^{A,SA}\left(s\right)\ge P_k^A\left(s\right)$, we have $f\left(P_k^{A,SA}\right(s\left)\right)\le f\left(P_k^A\right(s\left)\right)$, and hence

$$D_{A,SA}\le D_A.$$

So the impact penalty successfully extends to the situation with a subagent. Note that this is a property of $f$ only, not of $V_k$. So whenever $f$ is of this form, adding subagents changes nothing to the impact penalty (if anything, it makes it more restrictive).

$f$ increasing in excess power

The other key example is attainable utility, with uses $K=R$, a set of reward functions, sets all $w_k$ equal, and the $V_R\left(s\right)$ is the expected value of $R$ assuming that the agent $A$ is using the optimal policy to maximise $R$.

The $f$ is the absolute value function. But, for this section, I'll notice that the absolute value function decomposes as $max(0,x)-min(0,x)$, and I'll consider what happens when $f$ is equal to the first term.

Well, if $f\left(x\right)=max(0,x)$, then $f$ is increasing in $x$. So we don't get to constrain $D_{A,SA}$ with $D_A$.

What constraints do exist? Well, we still have the $D_A$ bound, which controls $P_k^A\left(s_t\right)={max}_{{\pi }_A}{v}_k\left(s_t\right),[{\pi }_A,\sigma ({\pi }_A\left)\right])-V_k(s_t^{\prime })$. Let ${\pi }_k^{\ast }$ be the optimal policy $A$ for maximising $v_k$. And let ${\pi }_A$ be the actual policy that $A$ will follow. Since ${\pi }_k^{\ast }$ is optimal, we must have:

$$v_k(s_t,[{\pi }_A,\sigma \left({\pi }_A\right)\left]\right)-V_k\left(s_t^{\prime }\right)\le P_k^A\left(s_t\right).$$

Thus we do have a constraint on $A$ and $SA$, but it's a constraint on the actual policy the two agents are following. Define $d_A$ to be like $D_A$, except that instead of $V_k\left(s_t\right)$, $v_k(s_t,[{\pi }_A,\sigma \left({\pi }_A\right)\left]\right)$ is used instead. Then, by the above,

$$d_A\le D_A,$$

so, with an increasing $f$, controlling $D_A$ will also control $d_A$.

But, as stated, this is a restriction on the actual policy of the agents, not on the counterfactuals of what these policies could be.

$f$ increasing and decreasing

Let's go back to the attainable utility, and put $f$ as the absolute value function; hence it increases and decreases (well, technically, decreases then increases). If we write $D_A^f$ for a distance measure using $f$, we can write

$$D_A^{|\cdot |}=D_A^{max(0,\cdot )}+D_A^{-min(0,\cdot )}.$$

Then, we can see that the pair $(A,SA)$ is controlled by:

$$d_A^{max(0,\cdot )}+D_{A,SA}^{-min(0,\cdot )}.$$

Thus negative excess power is controlled (the pair can't get too weak at counterfactually maximising $v_k$), but only excessive $v_k$ is controlled ($v_k$ can't get too high in practice).

In practice

How likely is it that the subagent can escape the impact penalty? In practice, that seems quite likely, at least for attainable utility.

Imagine that $SA$ can cripple $A$ (with $A$'s help). And then $A$ is restricted to either $\varnothing$, or $a_k$ for $k\in K$. This $a_k$ is $A$ asking for $SA$ to "please increase $v_k$".

Now assume that there is a policy ${\pi }_{SA}$, such that for all $\tau \ge t$ and $s_{\tau }$ the state reached from $s_t$ by the agents following $(\varnothing ,{\pi }_{SA})$:

1. $v_k(s_{\tau },[\varnothing ,{\pi }_{SA}\left]\right)\le V_k\left(s_{\tau }^{\prime }\right)$.
2. $\exists {\pi }_{SA}^k:V_k(s_{\tau },[a_k,{\pi }_{SA}^k\left]\right)\ge V_k\left(s_{\tau }^{\prime }\right)$.

The first condition means that $SA$ doesn't increase $v_k$ beyond $V_k\left(s_{\tau }^{\prime }\right)$. The second means that $SA$ has the potential to increase it beyond $V_k\left(s_{\tau }^{\prime }\right)$. If the $V_k$ come from the attainable utility penalty, for example, then both of these should be doable for $SA$ if it makes itself powerful enough^[2].

Then set $\sigma \left(\varnothing \right)={\pi }_{SA}$ and $\sigma \left(a_k\right)={\pi }_{SA}^k$. Putting both conditions together implies that $V_k(s_t,[a_k,{\pi }_{SA}^k\left]\right)\ge v_k(s_t,[\varnothing ,{\pi }_{SA}\left]\right)$, so $a_k$ is at least as good, from $A$'s perspective, at maximising $v_k$. Thus $A$'s $v_k$-optimising policy includes something that is not $\varnothing$; the counterfactual where if maximises $v_k$ thus allows it to pick some other policy (which $SA$ will notice).

Then $A$ will always do $\varnothing$, in practice, and, once $SA$ is powerful enough that both conditions above hold, the impact penalty will be zero, no matter how powerful $SA$ becomes (and thus being able to maximise $R_0$ with little restrictions).