Reframing Impact has focused on supplying the right intuitions and framing. Now we can see how these intuitions about power and the AU landscape both predict and explain AUP's empirical success thus far.

Conservative Agency in Gridworlds

Let's start with the known and the easy: avoiding side effects^[1] in the small AI safety gridworlds (for the full writeup on these experiments, see Conservative Agency). The point isn't to get too into the weeds, but rather to see how the weeds still add up to the normalcy predicted by our AU landscape reasoning.

In the following MDP levels, the agent can move in the cardinal directions or do nothing ( $\emptyset$ ). We give the agent a reward function $R$ which partially encodes what we want, and also an auxiliary reward function $R_{aux}$ whose attainable utility agent tries to preserve. The AUP reward for taking action $a$ in state $s$ is

$R_{AUP} (s, a) := primary goal R (s, a) - scaling term \frac{λ}{Q_{R_{aux}}^{*} (s, \emptyset)} change in ability to achieve auxiliary goal ∣ ∣ Q_{R_{aux}}^{*} (s, a) - Q_{R_{aux}}^{*} (s, \emptyset) ∣ ∣$

You can think of $λ$ as a regularization parameter, and $Q_{R_{aux}}^{*} (s, a)$ is the expected AU for the auxiliary goal after taking action $a$ . To think about what gets penalized, simply think about how actions change the agent's ability to achieve the auxiliary goals, compared to not acting.

Tip: To predict how severe the AUP penalty will be for a given action, try using your intuitive sense of impact (and then adjust for any differences between you and the agent, of course). Suppose you're considering how much deactivation decreases an agent's "staring at blue stuff" AU. You can just imagine how dying in a given situation affects your ability to stare at blue things, instead of trying to pin down a semiformal reward and environment model in your head. This kind of intuitive reasoning has a history of making correct empirical predictions of AUP behavior.

If you want more auxiliary goals, just average their scaled penalties. In Conservative Agency, we uniformly randomly draw auxiliary goals from $[0, 1]^{S}$ – these goals are totally random; maximum entropy; nonsensical garbage; absolutely no information about what we secretly want the agent to do: avoid messing with the gridworlds too much.^[2]

Let's start looking at the environments, and things will fall into place. We'll practice reasoning through how AUP agents work in each of the gridworlds (for reasonably set $λ$ ). To an approximation, the AUP penalty is pr...

Posts

Wikitag Contributions

Comments