I would term "hope for " rather than "reliability", because it's about willingness to enact in response to belief in , but if is no good, you shouldn't do that. Indeed, for bad , having the property of is harmful fatalism, following along with destiny rather than choosing it. In those cases, you might want to or something, though that only prevents from being believed, that you won't need to face in actuality, it doesn't prevent the actual . So reflects a value judgement about reflected in agent's policy, something downstream of endorsement of , a law of how the content of the world behaves according to an embedded agent's will.
Payor's Lemma then talks about belief in hope , that is hope itself is exogenous and needs to be judged (endorsed or not). Which is reasonable for games, since what the coalition might hope for is not anyone's individual choice, the details of this hope couldn't have been hardcoded in any agent a priori and need to be negotiated during a decision that forms the coalition. A functional coalition should be willing to act on its own hope (which is again something we need to check for a new coalition, that might've already been the case for a singular agent), that is we need to check that is sufficient to motivate the coalition to actually . This is again a value judgement about whether this coalition's tentative aspirations, being a vehicle for hope that , are actually endorsed by it.
Thus I'd term "coordination" rather than "trust", the fact that this particular coalition would tentatively intend to coordinate on a hope for . Hope is a value judgement about , and in this case it's the coalition's hope, rather any one agent's hope, and the coalition is a temporary nascent agency thing that doesn't necessarily know what it wants yet. The coalition asks: "If we find ourselves hoping for together, will we act on it?" So we start with coordination about hope, seeing if this particular hope wants to settle as the coalition's actual values, and judging if it should by enacting if at least coordination on this particular hope is reached, which should happen only if is a good thing.
(One intuition pump with some limitations outside the provability formalism is treating as "probably ", perhaps according to what some prediction market tells you. If "probably " is enough to prompt you to enact , that's some kind of endorsement, and it's a push towards increasing the equilibrium-on-reflection value of probability of , pushing "probably " closer to reality. But if is terrible, then enacting it in response to its high probability is following along with self-fulfilling doom, rather doing what you can to push the equilibrium away from it.)
Löb's Theorem then says that if we merely endorse a belief by enacting the believed outcome, this is sufficient for the outcome to actually happen, a priori and without that belief yet being in evidence. And Payor's Lemma says that if we merely endorse a coalition's coordinated hope by enacting the hoped-for outcome, this is sufficient for the outcome to actually happen, a priori and without the coordination around that hope yet being in evidence. The use of Löb's Theorem or Payor's Lemma is that the condition (belief in , or coordination around hope for ) should help in making the endorsement, that is it should be easier to decide to if you already believe that , or if you already believe that your coalition is hoping for . For coordination, this is important because every agent can only unilaterally enact its own part in the joint policy, so it does need some kind of premise about the coalition's nature (in this case, about the coalition's tentative hope for what it aims to achieve) in order to endorse playing its part in the coalition's joint policy. It's easier to decide to sign an assurance contract than to unconditionally donate to a project, and the role of Payor's Lemma is to say that if everyone does sign the assurance contract, then the project will in fact get funded sufficiently.
Löb's Theorem:
In the following discussion, I'll say "reality" to mean , "belief" to mean , "reliability" to mean (ie, belief is reliable when belief implies reality), and "trust" to mean (belief-in-reliability).
Löb says that if you have trust, you have belief.
Payor says that if you can prove that trust implies reality, then you have belief.
So, both results give conditions for belief. Indeed, both results give conditions equivalent to belief, since in both cases the inference can also be reversed:
Furthermore, both results relate reliability with belief, through the intermediary of trust.
Löb is usually thought of as a negative statement, that you can only have trust when you already have belief. One explanation of this is that Löb is the converse of the "trivial" case of trust, where you derive trust from simple belief: . (Recall that this is the pivotal step 2 in the proof of Payor's lemma.) Löb simply reverses this; combining the two, we know that trust is synonymous with belief: . "The only case where we can have trust is the trivial case: belief."
But this doesn't totally dash hopes of productive use of trust, as the negative reading of Löb might suggest. In the decision-theoretic application of Löb, we can take advantage of Löb to help ensure preferred outcomes:
For some desirable proposition , we can arrange things so that (reliability), and provably so (trust). Applying Löb, we get (belief). But we already arranged for ; so now we get out of the deal. This can allow cooperative handshakes between logic-based agents.
We can turn this into a general decision procedure by searching for the best we can make real in this way. More precisely, we search for possible implications of each action, and take the action for which we can find the most desirable implication. So we search for the relative provability of , in the context of an assumption that we take an action; and if we like what we see, we make that context real by taking that action.
Admittedly, this is a weird and spooky way to describe a fairly intuitive algorithm (search for actions with good consequences); the point is to try and clarify the connection to Löb as best I can.
The big problem here is that none of the above degrades well under uncertainty. Reading Löb as "trust is synonymous with belief" gives us a hint that it can't apply to probabilistic reasoning; clearly, I can think "my belief about is very probably very accurate" without assigning a high probability to . (In particular, if I assign a very low probability to .)
This results in a practical problem of Löb-based cooperation being very fragile, particularly for the general decision procedure mentioned above. Results are very sensitive to the proof strength of the participants. Intuitively, it shouldn't matter much whether I use or as my base-level reasoning systems, if I assign high probability to the soundness of both of those systems. For Löb-based handshakes, the difference can make or break cooperation.
Payor's Lemma looks much more amenable to probabilistic generalization, so it would be interesting to formulate a Payor-based decision procedure analogously to the Löb-based decision procedure mentioned earlier.
Where Löb is applied by making imply for desirable , Payor's lemma is applied via connecting with . So where Löb suggests the strategy make (good) beliefs imply reality, Payor suggests the strategy make trust (in the good) imply reality.
Critch gave the example strategy "cooperate if you know that if everyone believed everyone was cooperating, then everyone would indeed cooperate". However, this is a fixed strategy for a known game (it requires a pre-existing definition of "cooperation"). How can we turn it into a general decision procedure which can be applied to a wide variety of decision problems?
Trying to force the analogy:
| Löb | Payor | |
|---|---|---|
| statement of theorem | ||
| condition for belief | ||
| what we want to arrange to be (provably) true, for good | ||
| condition of above | ||
| condition outcome on action | ||
| what proofs to search for (looking for best for any ) |
When is interpreted as provability, these two decision procedures appear to be closely related, since as I mentioned before. This might be a good sign; the Löb-based procedure works well for proof-based decision theory. Since this equivalence is itself based on Löb, however, it'll break down for the probabilistic case. I'm not yet sure how to derive an actual suggested decision procedure for the probabilistic case. I'm not even sure this is the right way to force the analogy. (I don't think it generalizes Critch's proposed strategy... but I'm not sure how one would do that without being given data on the other agents, which seems anti-naturalistic.)