Sometimes written "Loeb's Theorem" (because umlauts are tricky). This is a theorem about proofs of what is provable and how they interact with what is actually provable in ways that surprise some people.

This math result often comes up when attempting to formalize "an agent" or "a value system" as somehow related to "a set of axioms".

Often, when making such mental motions, one wants to take multi-agent interactions seriously, and make the game-theoretically provably endorsable actions "towards an axiom system" be somehow contingent on what that other axiom system might or might not be able to game-theoretically provably endorse.

You end up with proofs about proofs about proofs... and then, without care, the formal proof systems themselves might explode or might give agentically incoherent results on certain test cases.

Sometimes, in this research context, the phrase "loebstacle" or "Löbstacle" comes up. This was an area of major focus (and a common study guide pre-requisite) for MIRI from maybe 2011 to 2016?

It became much less important later after the invention/discovery of the Garrabrant Inductor.

As to the math of Löb's theorem itself...

We trust Peano Arithmetic to correctly capture certain features of the standard model of arithmetic. Furthermore, we know that Peano Arithmetic is expressive enough to talk about itself in meaningful ways. So it would certainly be great if Peano Arithmetic asserted what now is an intuition: that everything it proves is certainly true.

In formal notation, let $P r v$ stand for the standard provability predicate of $P A$ . Then, $P r v (T)$ is true if and only if there is a proof from the axioms and rules of inference of $P A$ of $T$ . Then what we would like $P A$ to say is that $P r v (S) ⟹ S$ for every sentence $S$ .

But alas, $P A$ suffers from a problem of self-trust.

Löb's theorem states that if $P A ⊢ P r v (S) ⟹ S$ then $P A ⊢ S$ . This immediately implies that if $P A$ is consistent, the sentences $P A ⊢ P r v (S) ⟹ S$ are not provable when $S$ is false, even though according to our intuitive understanding of the standard model every sentence of this form must be true.

Thus, $P A$ is incomplete, and fails to prove a particular set of sentences that would increase massively our confidence in it.

Notice that Gödel's second incompleteness theorem follows immediately f...