1.18 Information theory essentials
You can read entropy, cross-entropy and KL in a loss and know what is being measured.
Before:00. Orientation & SetupUnlocks:03. Data Handling & Analysis04. Classical AI — Agents, Search & Knowledge Representation
Entropy measures surprise, cross-entropy measures the cost of encoding one distribution with another, and KL divergence is the gap between them — three quantities that appear in classification losses, language-model evaluation and VAEs. It sits at the end of the probability run because it reads its predecessors' vocabulary fluently. The confusion that matters in practice is treating KL as a distance; it is asymmetric, and which direction you take changes what the optimisation cares about.
Work through these
Entropy and surprise
Entropy measures how surprising a distribution is on average, which is the same as how many bits are needed to describe outcomes from it. It is the base unit of everything else here.
Cross-entropy and its use as a loss
Cross-entropy measures the cost of using the wrong distribution to describe data, which is precisely what a misfit classifier does. That is why it is the standard classification loss.
KL divergence and its asymmetry
The divergence between two distributions is not symmetric, and which way round you write it changes what the resulting model does. This asymmetry has real consequences in generative modelling.
Mutual information and feature selection
Mutual information measures how much knowing one variable tells you about another, without assuming a linear relationship. It is a more honest dependence measure than correlation.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.