1.24 Bayesian inference in practice

What this is and why it exists

Bayesian inference is pr1's update rule promoted from events to MODELS: begin with a belief over parameters, feed in data, and hold a posterior — a full distribution of what remains plausible. This topic makes it practical: the conjugate cases you can do on paper, the two engines (MCMC, variational) that handle everything else, and the small real model in PyMC or NumPyro that turns philosophy into a fitted object with honest uncertainty.

The vocabulary

• Prior — belief over parameters before the data; honesty about what you assumed, stated where all can see it.
• Posterior — belief after: proportional to likelihood times prior (pr6's objects, kept whole instead of maximised).
• Conjugate prior — a prior family the likelihood returns unchanged in form: posterior arithmetic in closed form.
• Beta–binomial — THE worked example: a beta prior on a rate plus binomial data gives a beta posterior by simple count-updating.
• MCMC — Markov chain Monte Carlo: a guided random walk whose visits are distributed as the posterior; exact in the limit, costs compute.
• Variational inference (VI) — fit a tractable family to the posterior by optimisation (minimising a KL divergence — pr7's asymmetry, live); fast, approximate.
• Credible interval — a region holding 95% of the POSTERIOR: "the parameter is in here with 95% probability", said legitimately.

The mental model

The posterior is a compromise between prior and likelihood, weighted by information. The beta–binomial shows the whole mechanism in arithmetic a beginner can audit: hold a Beta(a, b) belief about a conversion rate — readable as "a successes and b failures already imagined" — observe s successes and f failures, and the posterior is Beta(a+s, b+f). Counts add; belief sharpens. With 3 sales in 10 trials and a mild prior, the posterior peaks near 0.3 but stays wide — the honest statement at n = 10; by n = 1000 the prior's pseudo-counts are a rounding error and the data has taken over. That prior-fades-as-data-grows behaviour is pr6's MLE-versus-MAP story with the full distributions kept, and keeping them is the point: decisions read richer questions off a posterior — P(rate above break-even), expected loss of shipping — than any single estimate answers.

Off the conjugate garden path, two engines. MCMC explores the posterior by a random walk designed to spend time in proportion to plausibility; modern samplers (Hamiltonian variants under Stan, PyMC, NumPyro) climb efficiently even in high dimensions. It is asymptotically exact and computationally honest about its cost, and its diagnostics — did the chains mix, do independent chains agree — are part of using it, not optional extras. VARIATIONAL inference trades exactness for speed: pick a friendly family, optimise its parameters to minimise KL divergence to the true posterior — pr7's mode-seeking direction, so VI typically lands tight on one mode and UNDERSTATES uncertainty; the standard trade when data is huge and wall-clock matters.

The practicum is deliberately small: a beta–binomial or a Bayesian two-group comparison in PyMC or NumPyro — define prior and likelihood in a dozen lines, press the sampler, and READ the posterior: plot it, take its 95% credible interval, answer one decision question from it. And say the interval's meaning out loud, because here the tempting sentence is finally true: given model and prior, the parameter lies in the credible interval with 95% probability — the statement st2's confidence interval could not make, purchasable at the price of a stated prior. That price-and-purchase is the honest one-line summary of the whole Bayesian trade.

What you should now be able to explain or do

Update a beta prior by hand and narrate the pseudo-count reading. Say when conjugacy applies and what MCMC and VI each cost and buy. Fit one small model in PyMC or NumPyro and extract a credible interval. State the credible-versus-confidence distinction in two sentences.

Check yourself