The last couple posts compared some specific models for 20000 rolls of a die. This post will step back, and talk about more general theory for Bayesian model comparison.

The main problem is to calculate $P [d a t a | m o d e l]$ for some model. The model will typically give the probability of observed data $x$ (e.g. die rolls) based on some unobserved parameter values $θ$ (e.g. the $p$ 's in the last two posts), along with a prior distribution over $θ$ . We then need to compute

$P [d a t a | m o d e l] = \int_{θ} P [d a t a | θ] d P [θ]$

which will be a hairy high-dimensional integral.

Some special model structures allow us to simplify the problem, typically by factoring the integral into a product of one-dimensional integrals. But in general, we need some method for approximating these integrals.

The two most common approximation methods used in practice are Laplace approximation around the maximum-likelihood point, and MCMC (see e.g. here for application of MCMC to Bayes factors). We'll mainly talk about Laplace approximation here - in practice MCMC mostly works well in the same cases, assuming the unobserved parameters are continuous.

Laplace Approximation

Here's the idea of Laplace approximation. First, posterior distributions tend to be very pointy. This is mainly because independent probabilities multiply, so probabilities tend to scale exponentially with the number of data points. Think of the probabilities we calculated in the last two posts, with values like $10^{- 70}$ or $10^{- 20}$ - that's the typical case. If we're integrating over a function with values like that, we can basically just pay attention to the region around the highest value - other regions will have exponentially small weight.

Laplace' trick is to use a second-order approximation within that high-valued region. Specifically, since probabilities naturally live on a log scale, we'll take a second order-approximation of the log likelihood around its maximum point. Thus:

$\int_{θ} e^{l n P [d a t a | θ]} d P [θ] \approx \int_{θ} e^{l n P [d a t a | θ_{m a x}] + \frac{1}{2} (θ - θ_{m a x})^{T} (\frac{d^{2} l n P}{d θ^{2}} |_{θ_{m a x}}) (θ - θ_{m a x})} d P [θ]$

If we assume that the prior $d P [θ]$ is uniform (i.e. $d P [θ] = d θ$ ), then this looks like a normal distribution on $θ$ with mean $θ_{m a x}$ and variance given by the inverse Hessian matrix of the log-likelihood. (It turns out that, even for non-uniform $d P [θ]$ , we can just transform $θ$ so that the prior looks uniform near $θ_{m a x}$ , and transform it back when we're done.) The result:

$\int_{θ} e^{l n P [}$ ...