A Heuristic Derivation of the BIC

This is a note to myself to help recall a heuristic derivation of the Bayesian Information Criterion taught me by Claude.

The probability of data $D$ under a model $M$ is found by integrating out its parameters, giving the marginal likelihood, also known as the model evidence, $$ p(D|M) = \int p(D | \theta, M) p(\theta|M) d\theta.$$

The heuristic behind the BIC is super simple.

1. We assume the prior over parameters is approximately constant, so $$p(\theta|M)\approx \Delta_\text{prior}^{-1}.$$
2. We assume that the posterior is unimodal, with peak at $\theta^*$ and corresponding likelihood $p(D|\theta^*, M)$.
3. We assume the width of the peak is $\Delta_\text{post}$.

Putting these together we get $$ p(D|M) \approx p(D|\theta^*, M) \underbrace{\Delta_\text{post} \over \Delta_\text{prior}}_\text{Occam factor}.$$

The first term is the likelihood and will tend to improve as we add more parameters.

The second term is the Occam factor and reflects the integration over the parameter space. How does it behave?

We can write our uncertainty as a product of contributions from each of the $k$ parameter dimensions. If we assume for simplicity that these are the same width in each direction (dropping the assumption just means indexing the dimensions), then $$ {\Delta_\text{post} \over \Delta_\text{prior}} \approx \left(\sigma_\text{post} \over \sigma_\text{prior}\right)^k.$$

Now $\sigma_\text{post}$ will tend to behave as $\sigma_1/\sqrt{N}$ where $\sigma_1$ is the posterior width after a single observation, and $N$ is the number of observations. Recall, for example, how the variance of an estimator of the mean of a random variable drops with observations. Then,

$$ {\Delta_\text{post} \over \Delta_\text{prior}} \approx \left(\sigma_1 \over \sigma_\text{prior} \sqrt{N} \right)^k.$$

Inserting this back into our approximation of the model evidence, we get $$ \log p(D|M) \approx L – {k \over 2} \log N + \text{(terms constant in $N$)},$$ where $$ L = \log p(D|\theta^*,M)$$ is the log likelihood of the best-fitting parameters.

The Bayesian Information Criterion then just takes the first two terms, doubles them to remove the factor ${1 \over 2}$, and sign inverts (so lower is better) to get $$ \text{BIC} = k \log N – 2 L.$$

Conceptually, as we add more parameters, $L$ will increase, as our model fits the data better, but as parameters increase further, we’ll be fitting noise. The Occam factor measures how precisely more complex models specify their parameters, penalizing overfitting as the ratio of the posterior volume over the prior, which decreases as parameters increase. Once we start fitting noise, this decrease in volume is not compensated by a sufficient improvement in the fit, and the BIC starts to flatten, or increase, producing an elbow, and the “optimal” parameter count.

$$ \blacksquare$$