We should be more careful, though, about what we mean by saying that $\phi \left(x\right)$ only depends on ${Tr}_m$ for $m>n$, though, since this cannot be a purely syntactic criterion if we allow quantification over the subscript (as I did here). I'm pretty sure that something can be worked out, but I'll leave it for the moment.

I would suggest changing this system by defining $\psi \left(n\right)$ to mean that no $m\le n$ is the Gödel number of a proof of an inconsistency in ZFC (instead of just asserting that $n$ isn't). The purpose of this is to make it so that if ZFC were inconsistent, then we only end up talking about a finite number of levels of truth predicate. More specifically, I'd define $T_n$ to be PA plus the axiom schema

$$\forall m\ge n.\psi \left(m\right)\to \forall x.{Tr}_m(┌\phi (\overline{x})┐)\leftrightarrow \phi \left(x\right).$$

Then, it seems that Jacob Hilton's proof that the waterfalls are consistent goes through for this waterfall:

Work in ZFC and assume that ZFC is inconsistent. Let $n$ be the lowest Gödel number of a proof of an inconsistency. Let $M$ be the following model of our language: Start with the standard model of PA; it remains to give interpretations of the truth predicates. If $m\ge n$, then ${Tr}_m\left(k\right)$ is false for all $k$. If $m<n$, then ${Tr}_m\left(k\right)$ is true iff $k$ is the Gödel number of a true formula involving only ${Tr}_{m^{\prime }}$ for $m^{\prime }>m$. Then, it's clear that $T_0$, and hence all $T_m$ (since $T_0$ is the strongest of the systems) is sound on $M$, and therefore consistent.

Thus, we have proven in ZFC that if ZFC is inconsistent, then $T_0$ is consistent; or equivalently, that if $T_0$ is inconsistent, then ZFC is consistent. Stepping out of ZFC, we can see that if $T_0$ is inconsistent, then ZFC proves this, and therefore in this case ZFC proves its own consistency, implying that it is inconsistent. Hence, if ZFC is consistent, then so is $T_0$.

(Moreover, we can formalize this reasoning in ZFC. Hence, we can prove in ZFC (i) that if ZFC is inconsistent, then $T_0$ is consistent, and (ii) that if ZFC is consistent, then $T_0$ is consistent. By the law of the excluded middle, ZFC proves that $T_0$ is consistent.)

Hm; we could add an uninterpreted predicate symbol $Q\left(n\right)$ to the language of arithmetic, and let $s\equiv Q\left(0\right)$ and $r_n\equiv Q\left(\overline{n+1}\right)$. Then, it seems like the only barrier to recursive enumerability of $T_{\infty }$ is that $P$'s opinions about $Q(\cdot )$ aren't computable; this seems worrying in practice, since it seems certain that we would like logical uncertainty to be able to reason about the values of computations that use more resources than we use to compute our own probability estimates. But on the other hand, all of this makes this sound like an issue of self-reference, which is its own can of worms (once we have a computable process assigning probabilities to the value of computations, we can consider the sentence saying "I'm assigned probability $<\frac{1}{2}$" etc.).

The other direction follows from the fact that the algorithm is bounded, and PA can simply show the execution trace of $A$ in $\le CN$ steps.

Unimportant technical point: I think the length of the PA proof grows faster than this. (More precisely, the length in symbols, rather than the length in number of statements; we're almost always interested in the former, since it determines how quickly a proof can be checked or be found by exhaustive search.) The obvious way of showing $A\left(\right)=1$ in PA is to successively show for higher and higher $t$ that "after $t$ ticks, the Turing machine computing $A\left(\right)$ is in the following state, and its tapes have the following content". But even to write out the contents of the tape of a Turing machine that has run for $t$ ticks will in general take $t$ symbols. So the obvious way of doing the proof takes at least $O\left(t^2\right)$ symbols. Moreover, I'm not sure whether we can get from "at time $t$, the configuration is such-and-such" to "at time $t+1$, the configuration is such-and-such" in a constant number of proof steps. I suspect we don't need more than $O\left(t^3\right)$ symbols overall, but I'm not extremely confident. (I'd be quite surprised if the time turned out not to be polynomial, though!)

Next, we consider the case that PA is consistent and work through the agent’s decision. PA can’t prove $A\left(\right)\ne 1$, since we used the chicken rule, so since the sentence $A\left(\right)=1\to U\left(\right)=5$ is easily provable, the sentence $A\left(\right)=1\to U\left(\right)=10$ (ie. the first sentence that the agents checks for proofs of) must be unprovable.

It seems like this argument needs soundness of PA, not just consistency of PA. Do you see a way to prove in PA that if $PA⊢A\left(\right)\ne 1$, then PA is inconsistent?

[edited to add:] However, your idea reminds me of my post on the odd counterfactuals of playing chicken, and I think the example I gave there makes your idea go through:

The scenario is that you get 10 if you take action 1 and it's not provable that you don't take action 1; you get 5 if you take action 2; and you get 0 if you take action 1 and it's provable that you don't. Clearly you should take action 1, but I prove that modal UDT actually takes action 2. To do so, I show that PA proves $A\left(\right)=1\to \neg \square ┌A\left(\right)=1┐$. (Then from the outside, $A\left(\right)=2$ follows from the outside by soundness of PA.)

This seems to make your argument go through if we can also show that PA doesn't show $A\left(\right)\ne 1$. But if it did, then modal UDT would take action 1 because this comes first in its proof search, contradiction.

Thus, PA proves $A\left(\right)=1\to U\left(\right)=0$ (because this follows from $A\left(\right)=1\to \neg \square ┌A\left(\right)=1┐$), and also PA doesn't prove $A\left(\right)=1\to U\left(\right)=10$. As in your argument, then, the trolljecture implies that we should think "if the agent takes action 1, it gets utility 0" is a good counterfactual, and we don't think that's true.

Still interested in whether you can make your argument go through in your case as well, especially if you can use the chicken step in a way I'm not seeing yet. Like Patrick, I'd encourage you to develop this into a post.

If you replace the inner "$=1$" by "$>1-ϵ$", then the literal thing you wrote follows from the reflection principle: Suppose that the outer probability is $<1$. Then

$$P\left[\right(a<P\left[\phi \right]<b)\wedge P[a-ϵ<P\left[\phi \right]<b+\epsilon ]\le 1-ϵ]>0.$$

Now, $P[a<P[\phi ]<b]>0$ implies $P[a\le P[\phi ]\le b]>0$, which by the converse of the outer reflection principle yields $a\le P\left[\phi \right]\le b$, whence $a-ϵ<P\left[\phi \right]<b+ϵ$. Now, by the forward direction of the outer reflection principle, we have

$$P[a-ϵ<P[\phi ]<b+ϵ]=1>1-ϵ,$$

which, by the outer reflection principle again, implies

$$P\left[P\right[a-ϵ<P\left[\phi \right]<b+ϵ]>1-ϵ]=1,$$

a contradiction to the assumption that $\cdots \le 1-ϵ$ had outer probability $>0$.

However, what we'd really like is an inner reflection principle that assigns probability one to the statement *quantified over all $a$, $b$, and Gödel numbers $┌\phi ┐$. I think Paul Christiano has a proof that this is impossible for small enough $ϵ$, but I don't remember how the details worked.

This is very interesting!

My main question, which I didn't see an answer to on my first read, is: If the agent proves that action $a$ leads to the goal $G$ being achieved, are there any conditions under which this implies that the goal actually is achieved? The procrastination paradox shows that this isn't true in general. Is there a class of sentences (e.g., all ${\Pi }_1$ sentences, though I don't expect that to be true in this case) such that if $PA\star ⊢\phi$ then $N\models \phi$? In other words, do we have some guarantee that we do better at actually achieving $G$ than an agent which uses the system $BAD:=PA+Con\left(BAD\right)$?

A more technical point: Given that you claim that Theorem 3.2 (reflectively coherent quantified belief) follows "trivially" from Theorem 2.4, you presumably mean Theorem 2.4 to state that $PA\star ⊢\forall x.\square ┌\phi \left(\overline{x}\right)┐\to \phi \left(x\right)$, rather than $PA\star ⊢\forall x.\square ┌\phi ┐\to \phi$. (Even if you consider $\phi$ to implicitly contain an occurrence of $x$, it is essential that in $\square ┌\phi \left(\overline{x}\right)┐$ the free variable is replaced by the numeral corresponding to the outer value of $x$.)

Categorization is hard! :-) I wanted to break it up because long lists are annoying to read, but there was certainly some arbitrariness in dividing it up. I've moved "resource gathering agent" to the odds & ends.

Want to echo Nate's points!

One particular thing that I wanted to emphasize is that I think you can see as a thread on this forum (in particular, the modal UDT work is relevant) is that it's useful to make formal toy models where the math is fully specified, so that you can prove theorems about what exactly an agent would do (or, sometimes, write a program that figures it out for you). When you write out things that explicitly, then, for example, it becomes clearer that you need to assume that a decision problem is "fair" (extensional) to get certain results, as Nate points out (or if you don't assume it, someone else can look at your result and point out that it's not true as stated). In your post, you're using "logical expectations" that condition on something being true, without defining exactly what all of this means, and as a result you can argue about what these agents will do, but not actually prove it; that's certainly a reasonable part of the research process, but I'd like to encourage you to turn your work into models that are fully specified, so that you can actually prove theorems about them.

Some additions about how the initial system is going to work:

• Non-member contributions (comments and links) are going to become publicly visible when they have received 2 likes (from members---only members can like things).

• However, members will be able to reply to a contribution as soon as it has received 1 like. This means that if you think someone's made a useful contribution, and you want to reply to them about it, you don't have to wait for a second person to Like it before you can write that reply. (It won't be publicly visible until the contribution has two Likes, though.)