The paper Optimal Policies Tend to Seek Power tries to justify the claim in the title with a mathematical argument, using a clever formal definition of "power". I like the general approach, but I think that parts of the formalisation do not correspond well to the intuitive understanding of the concepts involved. This wouldn't be an issue if the goal of the paper was to prove a mathematical theorem. Here, however, the goal is to be able to deduce something about the behaviour of future AI systems. Therefore, the interface between "math" and "the world" is much more important. This post attempts to take another route to formalisation, which I think is closer to the intuition, albeit more difficult to work with mathematically.

Intro

The power of a state $s$ , written $P O W E R_{μ} (s)$ , is defined as (roughly) the average optimal value of that state, over some chosen distribution of rewards $μ$ . The paper quite convincingly argues for it:

Power is the ability to achieve a range of goals. In an MDP, the optimal value function $V_{R}^{*} (s, γ)$ captures the agent’s ability to “achieve the goal” $R$ . Therefore, average optimal value captures the agent’s ability to achieve a range of goals.

The problem with this approach is that it's very sensitive to the choice of the distribution of rewards $μ$ . In particular, one might take a Dirac delta on a reward giving 1 in any chosen state, and 0 everywhere else, which would trivially assign maximal power to that particular state. To combat this difficulty, authors introduce some technical devices allowing them to say "for most distributions, the power of (one, obviously better, state) is greater than the power of (another, obviously worse, state)".

To me, " $X$ happens for most distributions of rewards" should mean that there is some reasonable, uninformative, broad prior $D$ over distributions of rewards, and that $P_{D} (X) > \frac{1}{2}$ . The approach chosen in the paper not only doesn't correspond to this intuitive understanding, but seems to me to be (in certain sense) proving the thesis by assuming the conclusion.

More concretely, the approach taken in the paper is as follows. Given a reward $R$ and a permutation $σ$ over states of the MDP, one can obtain another reward $(σ R) (s) = R (σ (s))$ by permuting the states' rewards. This also works if we have not one reward $R$ , but a distribution over rewards, by pushing it forward by $σ$ . Therefore, each such permutati...

Measure Theory

Measure Theory