1.17 Maximum likelihood and MAP

What this is and why it exists

Where do loss functions come from? Not from a menu — from a principle. Maximum likelihood says: choose the parameters under which the observed data was most probable, and the familiar losses fall out as theorems: squared error from Gaussian noise, cross-entropy from categorical outputs. Derive them once and you stop memorising losses and start choosing noise models — which is what senior people actually do.

The vocabulary

• Likelihood — the probability of the DATA as a function of the PARAMETERS: same formula as probability, opposite reading.
• MLE (maximum likelihood estimate) — the parameter value making the observed data most probable.
• Log-likelihood — the log of the same; sums replace products, argmax unchanged.
• Negative log-likelihood (NLL) — the log-likelihood flipped into a loss to minimise.
• Prior / posterior — belief about parameters before and after data (Bayes, from pr1, applied to parameters).
• MAP (maximum a posteriori) — the parameter maximising the posterior: likelihood times prior.
• Cross-entropy — the classification loss; identically the NLL of a categorical model.

The mental model

Likelihood is probability read backwards. P(data given θ) with the data FIXED and θ varying is a function over parameters — not a distribution over them (it need not integrate to one). MLE climbs it. Logs are taken because independent data multiplies: products of thousands of probabilities underflow and differentiate miserably; sums behave. Maximising log-likelihood equals minimising NLL — and that sign flip is the entire secret of "where losses come from".

Run the derivations once, slowly. Bernoulli coin, k heads in n flips: differentiate the log-likelihood, get k/n — the frequency, now a theorem instead of an instinct. Gaussian noise around a mean: the log-density is a negative squared distance, so maximising likelihood IS minimising squared error — least squares (la6) unmasked as "MLE under Gaussian noise", and suddenly la6 and this topic are one story. Linear model with Gaussian noise: same result, feature-weighted. Categorical output: the NLL of the true class's predicted probability is exactly cross-entropy — the classifier's loss was never a separate invention; it is negative log-likelihood wearing its own name. When someone asks "why cross-entropy?", the answer is one sentence: because the model's output is a categorical distribution and we are doing maximum likelihood.

MAP adds the prior back. Multiply likelihood by prior, maximise the product — and in logs, the prior becomes an ADDED PENALTY. A Gaussian prior on weights adds a squared-norm term: ridge regression (la6's L2) is MAP under a Gaussian prior. A Laplace prior adds an absolute-value term: lasso is MAP under Laplace. Regularization, third derivation: la6 saw it as geometry, ca5 as soft constraint, and here it is a belief that weights are probably small. Small data lets the prior matter (k/n says 0/5 flips means "impossible"; a mild prior says "unlikely"); as data grows, likelihood swamps prior and MLE and MAP converge — which is exactly how it should be.

What you should now be able to explain or do

Distinguish likelihood from probability in one sentence. Derive the Bernoulli and Gaussian MLEs on paper. Say why cross-entropy is NLL, and produce ridge and lasso as MAP with named priors. State when MAP and MLE disagree and why.

Check yourself