Summary: Formally defining Extinction-level Goodhart's Law is tricky, because formal environments don't contain any actual humans that could go extinct. But we can do it using the notion of an interpretation mapping, which sends outcomes in the abstract environment to outcomes in the real world. We can then the truth condition of Extinction-level Goodhart's Law as a property of the environment.
I conjecture that Extinction-level Goodhart's Law does not hold in easily formalisable environments^[1], even though it might hold in the real world. This seems like a (very) big deal for AI advocacy, since it suggests that the lack of rigorous arguments concerning AI risk (eg, math proofs) does not provide strong evidence for the safety of AI.

Semi-formal definition of Extinction-level Goodhart's Law

Informally, we can define the extinction-level^[2] variant of Goodhart's law as follows:

Definition (informal): The Weak Version^[3] of Extinction-level Goodhart's Law is the claim that: "Virtually any goal specification, pursued to the extreme, will result in the extinction of humanity."

The tricky part is how to translate this into a more rigorous definition that can be applied to formal environments.

Defining "extinction" in formal environments

However, applying this definition to formal environments is tricky, because it requires formally defining which of the abstract states qualify as "extinction of humanity". How can we get past this obstacle?

What we don't do: "extinction states" given by definition

A lazy way of defining extinction in abstract models would be to assume that we are given some such "extinction" states by definition. That is, if $Ω$ is the set of all possible states in the formal environment, we could assume that there is some set $Ω_{extinction} \subset Ω$ . And we would refer to any $ω \in Ω_{extinction}$ as "extinction".
I don't like this approach, because it just hides the problem elsewhere. Also, this approach does not put any constraints on how $Ω_{extinction}$ should look like. As a result, we would be unlikely to be able to derive any results.

The approach we take instead is to augment the formal environment by an interpretation, where each state "abstract" state $ω$ in the formal environment $Ω$ is mapped onto some state $φ (ω)$ in some "more complex" environment $Ω^{'}$ .^[4]

Real-world interpretations

The typical use ...

Posts

Wikitag Contributions

Comments