In equation form, the AI is maximising

for some constant , some unlikely event that the AI cannot affect, some set of relevant descriptors , and some utility . Since C is constant, this is exactly the same as maximising - the probability is irrelevant.

The whole setup described is simply a way to ensure that if is the likely set of worlds consistent with observations after , then

(we "know" that doesn't happen and that we end up in ),

while

(in the worlds it cares about, the AI behaves as if was incredibly unlikely to come about).

New Comment