1.18 Information theory essentials

What this is and why it exists

Entropy, cross-entropy, KL — the three names haunt every loss function and every model card, and they measure one thing from three angles: surprise. This topic makes each a number you can compute and a sentence you can say, so that reading "minimises the KL divergence" in a paper costs you nothing.

The vocabulary

• Surprise — the negative log-probability of an outcome: rare events carry many bits, certainties carry none.
• Entropy H(p) — the average surprise of a distribution; its irreducible unpredictability, and the floor on lossless compression.
• Cross-entropy H(p, q) — the average surprise when reality follows p but you PRICED events by q; always at least H(p).
• KL divergence — the gap: cross-entropy minus entropy — the surprise you pay purely for modelling p as q. Zero only when they match; NOT symmetric.
• Mutual information — how many bits knowing one variable removes from the uncertainty of another; dependence measured in bits.

The mental model

Start with one outcome: its surprise is the negative log of its probability. A certain event: zero bits. A one-in-a-million event: about twenty bits of surprise. Entropy is the EXPECTED surprise of a source — a fair coin carries one bit per toss, a loaded 99:1 coin much less, and that number is simultaneously the floor on compressing the source's output: unpredictability and incompressibility are the same quantity, which is Shannon's founding insight.

Cross-entropy is surprise under the WRONG price list. Reality draws from p; your model quotes probabilities q; your average surprise is H(p, q) — always at least the entropy, because pricing by anything but the truth wastes bits. Now the classifier training loop reads itself aloud: the labels are (approximately) p, the network's softmax is q, and minimising cross-entropy is minimising your surprise at the data — pr6's NLL story retold in information's vocabulary. The two derivations are one loss.

KL is the pure penalty: H(p, q) minus H(p) — the bits paid for the MISMATCH alone, zero exactly at q = p, positive otherwise. And its asymmetry is a modelling decision, not a defect. KL(p‖q) charges you where p has mass and q says "impossible" — so minimising it makes q COVER everything p does (mode-covering). KL(q‖p) charges where q puts mass that p lacks — so minimising it makes q stay inside p's support (mode-seeking), happily latching onto one mode of a multimodal truth. Variational inference (st6) and various generative models differ in exactly which direction they minimise, and now you can read that choice as a stance.

Mutual information closes the set: I(X;Y) is the KL divergence between the joint and the product of marginals — the bits of dependence, linear or not, which is what correlation (pr3) could not hear. Feature selection by mutual information asks each feature: how many bits do you actually tell me about the label?

What you should now be able to explain or do

Compute entropy for small distributions and say what the number means twice (unpredictability, compression floor). Read a classifier's loss as cross-entropy between labels and softmax. Explain KL's asymmetry with the covering/seeking picture. Define mutual information and its edge over correlation.

Check yourself