Probability as Minimal Map argued that the probability $P [q | X]$ is a minimal representation of the information in data $X$ which is relevant to the query $q$ . In other words, $P [q | X]$ is a perfectly efficient map of some territory based on data $X$ , suitable for the query $q$ . More generally, a full probability distribution $P [Q | X]$ is a perfectly efficient map suitable for any queries about the random variable $Q$ . Bayesian probability tells how to update our map as we gain new information.

What if, for some reason, we wanted to instead throw away some particular information? We still want to represent our map using a probability distribution, but rather than adding information to it (via Bayes’ Rule), we want to remove some information.

Let’s start with an artificial but concrete example.

Coin-Sum Problem

I flip two coins, and observe both outcomes - either 0 or 1 for each. I want to keep as much information as possible, while throwing away all information from the observation relevant to their sum. How should I “update” my distribution over the outcomes?

We’ll write the outcome of our coinflip as $B = (B_{1}, B_{2})$ (“B” for “bit” or “binary” or “Bernoulli”), and our final information-removed distribution as $P [B | B, / \sum B]$ - so the notation $/ \sum B$ indicates that we should throw out all info about the sum (the “/” is meant to evoke e.g. a group quotient operation). Note that this kind of “remove information” operation does not follow the usual rules of Bayesian probability.

At first, I reason as follows:

My final distribution is a function of the outcomes I saw, so my overall strategy should specify four different distributions - one each for (0, 0), (1, 0), (0, 1), and (1, 1). E.g. if I see (0, 0), then I will update to the (0, 0) distribution $P [B | B = (0, 0), / \sum B]$
The final distribution for (0, 0) and (1, 1) must be the same, otherwise we could distinguish a sum of 0 from a sum of 2 by looking at which final distribution was chosen.
… but then the final distributions for (0, 1) and (1, 0) must also be the same as (0, 0), else we could tell that the sum is 1 whenever we see some other distribution.

… so in order to throw out all information about the sum, we have to throw out all the information from the observations. We effectively don’t update at all, and just keep our prior.

This seems kind of weird from an information-theoretic standpoint. We have 2 bits of information from the observation, and the...

Information Theory