In the previous article in this sequence, I conducted a thought experiment in which simple probability was not sufficient to choose how to act. Rationality required reasoning about meta-probabilities, the probabilities of probabilities.

Relatedly, lukeprog has a brief post that explains how this matters; a long article by HoldenKarnofsky makes meta-probability central to utilitarian estimates of the effectiveness of charitable giving; and Jonathan_Lee, in a reply to that, has used the same framework I presented.

In my previous article, I ran thought experiments that presented you with various colored boxes you could put coins in, gambling with uncertain odds.

The last box I showed you was blue. I explained that it had a fixed but unknown probability of a twofold payout, uniformly distributed between 0 and 0.9. The overall probability of a payout was 0.45, so the expectation value for gambling was 0.9—a bad bet. Yet your optimal strategy was to gamble a bit to figure out whether the odds were good or bad.

Let’s continue the experiment. I hand you a black box, shaped rather differently from the others. Its sealed faceplate is carved with runic inscriptions and eldritch figures. “I find this one particularly interesting,” I say.

What is the payout probability? What is your optimal strategy?

In the framework of the previous article, you have no knowledge about the insides of the box. So, as with the “sportsball” case I analyzed there, your meta-probability curve is flat from 0 to 1.

The blue box also has a flat meta-probability curve; but these two cases are very different. For the blue box, you know that the curve really is flat. For the black box, you have no clue what the shape of even the meta-probability curve is.

The relationship between the blue and black boxes is the same as that between the coin flip and sportsball—except at the meta level!

So if we’re going on in this style, we need to look at the distribution of probabilities of probabilities of probabilities. The blue box has a sharp peak in its meta-meta-probability (around flatness), whereas the black box has a flat meta-meta-probability.

You ought now to be a little uneasy. We are putting epicycles on epicycles. An infinite regress threatens.

Maybe at this point you suddenly reconsider the blue box… I told you that its meta-probability was uniform. But perhaps I was lying! How reliable do you think I am?

Let’s say you think there’s a 0.8 probability that I told the truth. That’s the meta-meta-probability of a flat meta-probability. In the worst case, the actual payout probability is 0, so the average just plain probability is 0.8 x 0.45 = 0.36. You can feed that worst case into your decision analysis. It won’t drastically change the optimal policy; you’ll just quit a bit earlier than if you were entirely confident that the meta-probability distribution was uniform.

To get this really right, you ought to make a best guess at the meta-meta-probability curve. It’s not just 0.8 of a uniform probability distribution, and 0.2 of zero payout. That’s the worst case. Even if I’m lying, I might give you better than zero odds. How much better? What’s your confidence in your meta-meta-probability curve? Ought you to draw a meta-meta-meta-probability curve? Yikes!

Meanwhile… that black box is rather sinister. Seeing it makes you wonder. What if I rigged the blue box so there is a small probability that when you put a coin in, it jabs you with a poison dart, and you die horribly?

Apparently a zero payout is not the worst case, after all! On the other hand, this seems paranoid. I’m odd, but probably not that evil.

Still, what about the black box? You realize now that it could do anything.

It might spring open to reveal a collection of fossil trilobites.
It might play Corvus Corax’s Vitium in Opere at ear-splitting volume.
It might analyze the trace DNA you left on the coin and use it to write you a personalized love poem.
It might emit a strip of paper with a recipe for dundun noodles written in Chinese.
It might sprout six mechanical legs and jump into your lap.

What is the probability of its giving you $2?

That no longer seems quite so relevant. In fact… it might be utterly meaningless! This is now a situation of radical uncertainty.

What is your optimal strategy?

I’ll answer that later in this sequence. You might like to figure it out for yourself now, though.

Further reading

The black box is an instance of Knightian uncertainty. That’s a catch-all category for any type of uncertainty that can’t usefully be modeled in terms of probability (or meta-probability!), because you can’t make meaningful probability estimates. Calling it “Knightian” doesn’t help solve the problem, because there’s lots of sources of non-probabilistic uncertainty. However, it’s useful to know that there’s a literature on this.

The blue box is closely related to Ellsberg’s paradox, which combines probability with Knightian uncertainty. Interestingly, it was invented by the same Daniel Ellsberg who released the Pentagon Papers in 1971. I wonder how his work in decision theory might have affected his decision to leak the Papers?