1.16 Law of large numbers and the CLT

What this is and why it exists

Why does averaging work? Why does the noisy mini-batch gradient point roughly the right way, and why do polls of 2000 people say anything about a country? Two theorems carry all of it: the law of large numbers (averages settle) and the central limit theorem (their errors are Gaussian, shrinking like one over root n). They are the licence statistics runs on — and knowing where the licence expires matters as much as holding it.

The vocabulary

• Law of large numbers (LLN) — the sample average of i.i.d. draws converges to the true expectation as n grows; the weak and strong forms differ in the technical sense of "converges".
• Central limit theorem (CLT) — the properly scaled average of many i.i.d. finite-variance draws approaches a normal distribution — whatever the draws' own distribution.
• Sampling distribution — the distribution of a statistic (like the mean) across imagined repeated samples; the object error bars describe.
• Standard error — the sampling distribution's standard deviation: σ over root n for a mean.
• i.i.d. — independent and identically distributed; the fine print both theorems stand on.

The mental model

The LLN is cancellation at scale: each draw errs above or below the truth, and with independence the errors have no conspiracy — they cancel, and the average homes in. That is why casinos are calm businesses, why insurance works, and why the mini-batch gradient (ca4) is trustworthy ON AVERAGE: it is a sample mean of per-example gradients, and the LLN is its warranty.

The CLT says more: it gives the ERROR a shape. Sum many small independent contributions — scale properly — and the result is bell-curved, regardless of what the pieces looked like: skewed, lumpy, discrete, anything with finite variance. Simulate it once and believe it forever: draw from a wildly skewed distribution, average 50 at a time, histogram the averages — a clean bell. This universality is why the Gaussian is everywhere (noise is summed micro-causes — the ECE unit's thermal noise says the same), and the root-n scaling is the economics of data: the standard error of a mean shrinks as σ over root n, so each halving of the error bar costs QUADRUPLE the data. Precision is bought at a steeply rising price — the single most budget-relevant formula in statistics (st2 spends it).

Where the licence expires — three honest clauses. Heavy tails: infinite-variance distributions break the CLT outright, and merely heavy tails make convergence slow — the bell emerges only at enormous n while the extremes still behave wildly (pr3's kurtosis warning, now with a mechanism). Dependence: correlated draws cancel less than independence promises — time series, clustered users and students within one college all deliver less information than their count suggests, so the EFFECTIVE n is smaller than the counted n. And smallness itself: the CLT is an n-grows promise; at n of 10 the bell may be a caricature. The professional reflex: before leaning on an average's error bar, ask about tails, dependence and n.

What you should now be able to explain or do

State both theorems informally and what each licenses. Explain the root-n law and its cost implication. Run (or narrate) the simulation that shows the CLT happening. Name the three failure clauses and one real dataset that trips each.

Check yourself