This is the appendix to the previous post on Goodhart’s Law and KL regularization, containing all of our proofs.

Theorem about distributions

Theorem 1: Given any heavy-tailed reference distribution $Q$ over $R$ with mean $μ_{Q}$ , and any $M, ϵ > 0$ , there is a distribution $P$ with mean $μ_{P} > M$ and $D_{K L} (P ∥ Q) < ϵ$ .

Proof: WLOG let $μ_{Q} = 0$ . We construct a sequence of distributions ${P_{t}}$ such that ${lim}_{t \to \infty} E_{P_{t}} [X] \geq c$ for any constant $c$ , and ${lim}_{t \to \infty} D_{K L} (P_{t} ∥ Q) = 0$ . We define $P_{t}$ for any $t > c$ thusly. Writing $F_{P_{t}} (x)$ for the CDF $P r_{X \sim P_{t}} (X \leq x)$ and ${¯ F}_{P_{t}} (x)$ for $1 - F_{P_{t}} (x)$ , we let

{¯ F}_{P_{t}} (x) = ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ \begin{matrix} 1 - \frac{1 - c / t}{F_{Q} (t)} F_{Q} (x) & x \leq t \frac{c / t}{{¯ F}_{Q} (t)} {¯ F}_{Q} (x) & x > t \end{matrix}

Intuitively, we rescale the part of the distribution to the right of $t$ evenly to have total probability $c / t$ , which is less than 1 because $t > c$ .

We must check that ${lim}_{t \to \infty} E_{P_{t}} [X] = c$ . We can write

\begin{matrix} E_{P_{t}} [X] = F_{P_{t}} (t) E_{P_{t}} [X | X \leq t] & + {¯ F}_{P_{t}} (t) E_{P_{t}} [X | X > t] = F_{P_{t}} (t) E_{Q} [X | X \leq t] & + {¯ F}_{P_{t}} (t) E_{Q} [X | X > t] = F_{Q} (t) E_{Q} [X | X \leq t] & + {¯ F}_{Q} (t) E_{Q} [X | X > t] + (F_{P_{t}} (t) - F_{Q} (t)) & E_{Q} [X | X \leq t] + ({¯ F}_{P_{t}} (t) - {¯ F}_{Q} (t)) E_{Q} [X | X > t] = E_{Q} [X] + ({¯ F}_{P_{t}} (t) - & {¯ F}_{Q} (t)) (E_{Q} [X | X > t] - E_{Q} [X | X \leq t]) \end{matrix}

We know that $E_{Q} [X | X > t] > t$ because it is an integral of values strictly greater than t. Because $E_{Q} [X] = 0$ is a weighted average of $E_{Q} [X | X > t]$ and $E_{Q} [X | X \leq t]$ , and $E_{Q} [X | X > t] > 0$ , we know $E_{Q} [X | X \leq t] < 0$ . So $E_{Q} [X | X > t] - E_{Q} [X | X \leq t] > t .$ We also know that for sufficiently large $t$ , $(F_{P_{t}} (t) - F_{Q} (t)) > 0$ . Intuitively, starting from $Q$ , which has mean 0, $P_{t}$ moves a probability mass approaching $\frac{c}{t}$ from mean <0 to mean >t.

Now we can say

lim t \to \infty

...

Catastrophic Regressional Goodhart