Due to the problem of building a strong $V$-enhancer when we want a $U$-enhancer - and the great difficulty in defining $U$, the utility we truly want to maximise - many people have suggested reducing the $V$-increasing focus of the AI. The idea is that, as long as the AI doesn't devote too much optimisation power to $V$, the $V$ and $U$ will stay connected with each other, and hence a moderate increase in $V$ will in fact lead to a moderate increase in $U$.

This has lead to interest in such things as satisficers and low-impact AIs, both of which have their problems. Those try and put an absolute limit on how much $V$ is optimised. The AI is not supposed to optimise $V$ above a certain limit (satisficer) or if optimising it changes too much about the world or the power of other agents (low-impact).

Another approach is to put a relative limit on how much an AI can push a utility function. For example, quantilizers will choose randomly among the top $0<q\le 1$ proportion of actions/policies, rather than picking the top action/policy. Then there is the approach of using pessimism to make the AI more conservative. This pessimism is defined by a parametre $\beta \in (0,1)$, with $\beta \to 1$ being very pessimistic.

Intermediate value uncertainty

The behaviours of $q$ and $\beta$ are pretty clear around the extremes. As $\beta$ and $q$ tend to $0$, the agent will behave like a $V$-maximiser. As they tend to $1$, the agent will behave randomly ($q$) or totally conservatively ($\beta$).

Thus, we expect that moving away from the extremes will improve the true $U$-performance, and that the conservative, $1$ end, will be less disastrous than the $V$-maximising, $0$ end (though we only know that second fact, because of implicit assumptions we have on $U$ and $V$).

The problem is in the middle, where the behaviour is unknown (and, since we lack a full formulation of $U$, generically unknowable). There is no principled way of setting the $q$ or the $\beta$. Consider, for example, this plot of $q$ versus $U$:

Here, the ideal $q$ is around $0.45$, but the critical thing is to keep $q$ above $0.3$: that's the point at which it falls precipitously.

Contrast now with this one:

Here, any value of $q$ above $0.2$ is essentially the same, and $q$ can be lowered as low as $0.05$ before there are any problems.

So, in the first case, we need $q$ above $0.3$, and, in the second, below $0.2$. And, moreover, it might be that the first situation appears in one world and the second in another, and both worlds are currently possible. So there's no consistent good value of $q$ we can set (and in the general case, the curve might be multi-modal, with many peaks). And note that we don't know any of these graphs (since we can't define $U$ fully). So we don't know what values to set $q$ at, have little practical guidance on what to do, but expect that some values will be disastrous.

The conservatism approach has similar problems: $\beta$ is even harder to interpret than $q$, we don't have any guidance on how to set it, and the ideal $\beta$ may vary considerably depending on the circumstance. For example, what would we want our AI to do when it finds an unexpected red button connected to nuclear weapons?

Well, that depends on whether the button starts a nuclear launch - or if it cancels one.

A future post will explore how to resolve this issue, and how to choose the conservatism parameter in a suitable way.