This is a more technical followup to the last post, putting precise bounds on when regressional Goodhart leads to failure or not. We'll first show conditions under which optimization for a proxy fails, and then some conditions under which it succeeds. (The second proof will be substantially easier.)

Related work

In addition to the related work sections of the previous post, this post makes reference to the textbook An Introduction to Heavy-Tailed and Subexponential Distributions, by Foss et al. Many similar results about random variables are present in the textbook, though we haven't seen this posts's results elsewhere in the literature before. We mostly adopt their notation here, and cite a few helpful lemmas.

Main result: Conditions for catastrophic Goodhart

Suppose that $X$ and $V$ are independent real-valued random variables. We're going to show, roughly, that if

$X$ is subexponential (a slightly stronger property than being heavy-tailed).
$V$ has lighter tails than $X$ by more than a linear factor, meaning that the ratio of the tails of $V$ and the tails of $X$ grows superlinearly.^[1]

then ${lim}_{t \to \infty} E [V | X + V \geq t] = E [V]$ .

Less formally, we're saying something like "if it requires relatively little selection pressure on $X$ to get more of $X$ and asymptotically more selection pressure on $V$ to get more of $V$ , then applying very strong optimization towards $X + V$ will not get you even a little bit of optimization towards $V$ - all the optimization power will go towards $X$ ."

Proof sketch and intuitions

The conditional expectation $E [V | X + V > t]$ is given by $\frac{\int_{- \infty}^{\infty} v f_{V} (v) Pr (X > t - v)}{\int_{- \infty}^{\infty} f_{V} (v) Pr (X > t - v)}$ ,^[2] and we divide the integral in the numerator into 4 regions, showing that each region's effect on the conditional expectation of V is similar to that of the corresponding region in the unconditional expectation $E [V]$ .

The regions are defined in terms of a slow-growing function $h (t) : R \to R_{\geq 0}$ such that the fiddly bounds on different pieces of the proof work out. Roughly, we want it to go to infinity so that $| V |$ is likely to be less than $h (t)$ in the limit, but grow slowly enough that the shape of $V$ 's distribution within the interval $[- h (t), h (t)]$ doesn't chan...

Catastrophic Regressional Goodhart