Some people think that they have unbounded utility functions. This isn't necessarily crazy, but it presents serious challenges to conventional decision theory. I think it probably leads to abandoning probability itself as a representation of uncertainty (or at least any hope of basing decision theory on such probabilities). This may seem like a drastic response, but we are talking about some pretty drastic inconsistencies.

This result is closely related to standard impossibility results in infinite ethics. I assume it has appeared in the philosophy literature, but I couldn't find it in the SEP entry on the St. Petersburg paradox so I'm posting it here. (Even if it's well known, I want something simple to link to.)

(ETA: this argument is extremely similar to Beckstead and Thomas' argument against Recklessness in A paradox for tiny probabilities and enormous values. The main difference is that they use transitivity +"recklessness" to get a contradiction whereas I argue directly from "non-timidity." I also end up violating a dominance principle which seems even more surprising to violate, but at this point it's kind of like splitting hairs. I give a slightly stronger set of arguments in Better impossibility results for unbounded utilities.)

Weak version

We'll think of preferences as relations over probability distributions over some implicit space of outcomes $Ω$ (and we'll identify outcomes with the constant probability distribution). We'll show that there is no relation $<$ which satisfies three properties: Antisymmetry, Unbounded Utilities, and Dominance.

Note that we assume nothing about the existence of an underlying utility function. We don't even assume that the preference relation is complete or transitive.

The properties

Antisymmetry: It's never the case that both $A < B$ and $B < A$ .

Unbounded Utilities: there is an infinite sequence of outcomes $X_{1}, X_{2}, X_{4}, X_{8}, \dots$ each "more than twice as good" as the last.^[1] More formally, there exists an outcome $X_{0}$ such that:

$X_{2^{k}} < \frac{1}{2} X_{2^{k + 1}} + \frac{1}{2} X_{0}$ for every $k$ .
$X_{0} < \frac{1}{2} X_{1} + \frac{1}{2} X_{0}$ ^[2]

That is, $X_{1}$ is not as good as a $\frac{1}{2}$ chance of $X_{2}$ , which is not as good as a $\frac{1}{4}$ chance of $X_{4}$ , which is not as good as a $\frac{1}{8}$ chance of $X_{8}$ ... This is nearly the weakest possible version of unbounded utilities.^[3]

Dominance: let $A_{0}, A_{1}, \dots$ and $B_{0}, B_{1}, \dots$ be sequences of lotteries, and $p_{0}, p_{1}, \dots$ be a sequence of probabilities that sum to 1. If $A_{i} < B_{i}$ for all $i$ , then $\sum_{i} p_{i} A_{i} < \sum_{i} p_{i} B_{i}$ .

Inconsistency proof

Consider the lottery $X_{\infty} = \frac{1}{2} X_{0} + \frac{1}{4} X_{1} + \frac{1}{8} X_{2} + \frac{1}{16} X_{4} + \dots$

We can write $X_{\infty}$ as a mixture:

$X_{\infty} = \frac{1}{2} (\frac{1}{2} X_{0} + \frac{1}{2} X_{1}) + \frac{1}{4} (\frac{1}{2} X_{0} + \frac{1}{2} X_{2}) + \frac{1}{8} (\frac{1}{2} X_{0} + \frac{1}{2} X_{4}) + \dots$

By definition $X_{0} < \frac{1}{2} X_{0} + \frac{1}{2} X_{1}$ . And for each $k$ , Unbounded Utilities implies that $X_{2^{k}} < \frac{1}{2} X_{0} + \frac{1}{2} X_{2^{k + 1}}$ . Thus Dominance implies $X_{\infty} < X_{\infty}$ , contradicting Antisymmetry.

How to avoid the paradox?

By far the easiest way out is to reject Unbounded Utilities. But that's just a statement about our preferences, so it's not clear we get to "reject" it.

Another common way out is to assume that any two "infinitely good" outcomes are incomparable, and therefore to reject Dominance.^[4] This results in being indifferent to receiving $1 in every world (if the expectation is already infinite), or doubling the probability of all good worlds, which seems pretty unsatisfying.

Another option is to simply ignore small probabilities, which again leads to rejecting even the finite version of Dominance---sometimes when you mix together lotteries something will fall below the "ignore it" threshold leading the direction of your preference to reverse. I think this is pretty bizarre behavior, and in general ignoring small probabilities is much less appealing than rejecting Unbounded Utilities.

All of these options seem pretty bad to me. But in the next section, we'll show that if the unbounded utilities are symmetric---if there are both arbitrarily good and arbitrarily bad outcomes---then things get even worse.

Strong version

I expect this argument is also known in the literature; but I don't feel like people around LW usually grapple with exactly how bad it gets.

In this section we'll show there is no relation $<$ which satisfies three properties: Antisymmetry, Symmetric Unbounded Utilities, and Weak Dominance.

(ETA: actually I think that even with only positive utilities you already violate something very close to Weak Dominance, which Beckstead and Thomas call Prospect-Outcome dominance. I find this version of Weak Dominance slightly more compelling, but Symmetric Unbounded Utilities is a much stronger assumption than Unbounded Utilities or non-Timidity, so it's probably worth being aware of both versions. In a footnote^[5] I also define an even weaker dominance principle that we are forced to violate.)

The properties

Antisymmetry: It's never the case that both $A > B$ and $B > A$ .

Symmetric Unbounded Utilities. There is an infinite sequence of outcomes $X_{1}, X_{- 2}, X_{4}, X_{- 8}, \dots$ each of which is "more than twice as important" as the last but with opposite sign. More formally, there is an outcome $X_{0}$ such that:

$X_{0} < X_{1}$
For every even $k$ : $\frac{1}{3} X_{- 2^{k + 1}} + \frac{2}{3} X_{2^{k}} < X_{0}$
For every odd $k$ : $\frac{1}{3} X_{2^{k + 1}} + \frac{2}{3} X_{- 2^{k}} > X_{0}$

That is, a certainty of $X_{1}$ is outweighed by a $\frac{1}{2}$ chance of $X_{- 2}$ , which is outweighed by a $\frac{1}{4}$ chance of $X_{4}$ , which is outweighed by a $\frac{1}{8}$ chance of $X_{- 8}$ ....

Weak Dominance.^[5] For any outcome $X$ , any sequence of lotteries $B_{0}, B_{1}, \dots$ , and any sequence of probabilities $p_{0}, p_{1}, \dots$ that sum to 1:

If $X < B_{i}$ for every $i$ , then $X < \sum p_{i} B_{i}$ .
If $X > B_{i}$ for every $i$ , then $X > \sum p_{i} B_{i}$ .

Inconsistency proof

Now consider the lottery $X_{\pm \infty} = \frac{1}{2} X_{1} + \frac{1}{4} X_{- 2} + \frac{1}{8} X_{4} + \frac{1}{16} X_{- 8} + \frac{1}{32} X_{16} + \dots$

We can write $X_{\pm \infty}$ as the mixture:

$\frac{3}{4} (\frac{2}{3} X_{1} + \frac{1}{3} X_{- 2}) + \frac{3}{16} (\frac{2}{3} X_{4} + \frac{1}{3} X_{- 8}) + \frac{3}{64} (\frac{2}{3} X_{16} + \frac{1}{3} X_{- 32}) \dots$

By Unbounded Utilities each of these terms is $< X_{0}$ . So by Weak Dominance, $X_{\pm \infty} < X_{0} .$

But we can also write $X_{\pm \infty}$ as the mixture:

$\frac{1}{2} X_{1} + \frac{3}{8} (\frac{2}{3} X_{- 2} + \frac{1}{3} X_{4}) + \frac{3}{32} (\frac{2}{3} X_{- 8} + \frac{1}{3} X_{16}) + \dots$

By Unbounded Utilities each of these terms is $> X_{0}$ . So by Weak Dominance $X_{\pm \infty} > X_{0}$ . This contradicts Antisymmetry.

Now what?

As usual, the easiest way out is to abandon Unbounded Utilities. But if that's just the way you feel about extreme outcomes, then you're in a sticky situation.

You could allow for unbounded utilities as long as they only go in one direction. For example, you might be open to the possibility of arbitrarily bad outcomes but not the possibility of arbitrarily good outcomes.^[6] But the asymmetric version of unbounded utilities doesn't seem very intuitively appealing, and you still have to give up the ability to compare any two infinitely good outcomes (violating Dominance).

People like talking about extensions of the real numbers, but those don't help you avoid any of the contradictions above. For example, if you want to extend $<$ to a preference order over hyperreal lotteries, it's just even harder for it to be consistent.

Giving up on Weak Dominance seems pretty drastic. At that point you are talking about probability distributions, but I don't think you're really using them for decision theory---it's hard to think of a more fundamental axiom to violate. Other than Antisymmetry, which is your other option.

At this point I think the most appealing option, for someone committed to unbounded utilities, is actually much more drastic: I think you should give up on probabilities as an abstraction for describing uncertainty, and should not try to have a preference relation over lotteries at all.^[7] There are no ontologically fundamental lotteries to decide between, so this isn't necessarily so bad. Instead you can go back to talking directly about preferences over uncertain states of affairs, and build a totally different kind of machinery to understand or analyze those preferences.

ETA: replacing dominance

Since writing the above I've become more sympathetic to violations of Dominance and even Weak Dominance---it would be pretty jarring to give up on them, but I can at least imagine it. I still think violating "Very Weak Dominance"^[5] is pretty bad, but I don't think it captures the full weirdness of the situation.

So in this section I'll try to replace Weak Dominance by a principle I find even more robust: if I am indifferent between $X$ and any of the lotteries $A_{i}$ , then I'm also indifferent between X and any mixture of the lotteries $A_{i}$ . This isn't strictly weaker than Weak Dominance, but violating it feels even weirder to me. At any rate, it's another fairly strong impossibility result constraining unbounded utilities.

The properties

We'll work with a relation $\leq$ over lotteries. We write $A = B$ if both $A \leq B$ and $B \leq A$ . We write $A < B$ if $A \leq B$ but not $A = B$ . We'll show that $\leq$ can't satisfy four properties: Transitivity, Intermediate mixtures, Continuous symmetric unbounded utilities, and Indifference to homogeneous mixtures.

Intermediate mixtures. If $A < B$ , then $A < \frac{1}{2} A + \frac{1}{2} B < B$ .

Transitivity. If $A \leq B$ and $B \leq C$ then $A \leq C$ .

Continuous symmetric unbounded utilities. There is an infinite sequence of lotteries $X_{1}, X_{- 2}, X_{4}, X_{- 8}, \dots$ each of which is "exactly twice as important" as the last but with opposite sign. More formally, there is an outcome $X_{0}$ such that:

$X_{0} < X_{1}$
For every even $k$ : $\frac{1}{3} X_{- 2^{k + 1}} + \frac{2}{3} X_{2^{k}} = X_{0}$
For every odd $k$ : $\frac{1}{3} X_{2^{k + 1}} + \frac{2}{3} X_{- 2^{k}} = X_{0}$

That is, a certainty of $X_{1}$ is exactly offset by a $\frac{1}{2}$ chance of $X_{- 2}$ , which is exactly offset by a $\frac{1}{4}$ chance of $X_{4}$ , which is exactly offset by a $\frac{1}{8}$ chance of $X_{- 8}$ ....

Intuitively, this principle is kind of like symmetric unbounded utilities, but we assume that it's possible to dial down each of the outcomes in the sequence (perhaps by mixing it with $X_{0}$ ) until the inequalities become exact equalities.

Homogeneous mixtures. Let $X$ be an outcome, $A_{0}, A_{1}, A_{2}, \dots$ , a sequence of lotteries, and $p_{0}, p_{1}, p_{2}, \dots$ be a sequence of probabilities summing to 1. If $A_{i} = X$ for all $i$ , then $\sum p_{i} A_{i} = X$ .

Inconsistency proof

Consider the lottery $X_{\pm \infty} = \frac{1}{2} X_{1} + \frac{1}{4} X_{- 2} + \frac{1}{8} X_{4} + \frac{1}{16} X_{- 8} + \frac{1}{32} X_{16} + \dots$

We can write $X_{\pm \infty}$ as the mixture:

$\frac{3}{4} (\frac{2}{3} X_{1} + \frac{1}{3} X_{- 2}) + \frac{3}{16} (\frac{2}{3} X_{4} + \frac{1}{3} X_{- 8}) + \frac{3}{64} (\frac{2}{3} X_{16} + \frac{1}{3} X_{- 32}) \dots$

By Unbounded Utilities each of these terms is $= X_{0}$ . So by homogeneous mixtures, $X_{\pm \infty} = X_{0} .$

But we can also write $X_{\pm \infty}$ as the mixture:

$\frac{1}{2} X_{1} + \frac{3}{8} (\frac{2}{3} X_{- 2} + \frac{1}{3} X_{4}) + \frac{3}{32} (\frac{2}{3} X_{- 8} + \frac{1}{3} X_{16}) + \dots$

By Unbounded Utilities each of these terms other than the first is $= X_{0}$ . So by Homogenous Mixtures, the combination of all terms other than the first is $= X_{0}$ . Together with the fact that $X_{1} > X_{0}$ , Intermediate Mixtures and Transitivity imply $X_{\pm \infty} > X_{0}$ . But that contradicts $X_{\pm \infty} = X_{0}$ .

^{^}
Note that we could replace "more than twice as good" with "at least 0.00001% better" and obtain exactly the same result. You may find this modified version of the principle more appealing, and it is closer to non-timidity as defined in Beckstead and Thomas. Note that the modified principle implies the original by applying transitivity 100000 times, but you don't actually need to apply transitivity to get a contradiction, you can just apply Dominance to a different mixture.
^{^}
You may wonder why we don't just write $X_{0} < X_{1}$ . If we did this, we'd need to introduce an additional assumption that if $A < B$ , $p A + (1 - p) X_{0} < p B + (1 - p) X_{0}$ . This would be fine, but it seemed nicer to save some symbols and make a slightly weaker assumption.
^{^}
The only plausibly-weaker definition I see is to say that there are outcomes $X_{0} < X_{1}$ and an infinite sequence $X_{> 2}, X_{> 3}, . . .$ such that for all $n$ : $\frac{1}{n} X_{> n} + (1 - \frac{1}{n}) X_{0} > X_{1}$ . If we replaced the $>$ with $=$ then this would be stronger than our version, but with the inequality it's not actually sufficient for a paradox.

To see this, consider a universe with three outcomes $X_{0}, X_{1}, X_{\infty}$ and a preference order $<$ that always prefers lotteries with higher probability of $X_{\infty}$ and breaks ties using by preferring a higher probability of $X_{1}$ . This satisfies all of our other properties. It satisfies the weaker version of the axiom by taking $X_{> n} = X_{\infty}$ for all $n$ , and it wouldn't be crazy to say that it has "unbounded" utilities.
^{^}
For realistic agents who think unbounded utilities are possible, it seems like they should assign positive probability to encountering a St. Petersburg paradox such that all decisions have infinite expected utility. So this is quite a drastic thing to give up on. See also: Pascal's mugging.
^{^}
I find this principle pretty solid, but it's worth noting that the same inconsistency proof would work for the even weaker "Very Weak Dominance": for any pair of outcomes with $X_{0} < X_{1}$ , and any sequence of lotteries $B_{i}$ each strictly better than $X_{1}$ , any mixture of the $B_{i}$ should at least be strictly better than $X_{0}$ !
^{^}
Technically you can also violate Symmetric Unbalanced Utility while having both arbitrarily good and arbitrarily bad outcomes, as long as those outcomes aren't comparable to one another. For example, suppose that worlds have a real-valued amount of suffering and a real-valued amount of pleasure. Then we could have a lexical preference for minimizing expected suffering (considering all worlds with infinite expected suffering as incomparable), and try to maximize pleasure only as a tie-breaker (considering all worlds with infinite expected pleasure as incomparable).
^{^}
Instead you could keep probabilities but abandon infinite probability distributions. But at this point I'm not exactly sure what unbounded utilities means---if each decision involves only finitely many outcomes, then in what sense do all the other outcomes exist? Perhaps I may face infinitely many possible decisions, but each involves only finitely many outcomes? But then what am I to make of my parent's decisions while raising me, which affected my behavior in each of those infinitely many possible decisions? It seems like they face an infinite mixture of possible outcomes. Overall, it seems to me like giving up on infinitely big probability distributions implies giving up on the spirit of unbounded utilities, or else going down an even stranger road.