1.14 Expectation, variance and moments

What this is and why it exists

A distribution is a whole curve; decisions want a few numbers. Expectation and variance are the great compression — where the mass sits, how far it spreads — and their algebra (linearity above all) is the daily arithmetic of ML: expected loss, expected reward, variance of an estimator. Skewness and kurtosis are the honesty clauses: what the first two numbers hide.

The vocabulary

• Expectation E[X] — the probability-weighted average; the distribution's balance point.
• Linearity — E[aX + bY] = aE[X] + bE[Y], ALWAYS — independence not required; the most-used fact in the subject.
• Variance — expected squared distance from the mean; its square root, the standard deviation, restores the original units.
• Covariance / correlation — how two variables move together; correlation is covariance normalised into the range −1 to 1.
• Skewness — asymmetry: which tail is long.
• Kurtosis — tail weight: how much probability lives in extremes, relative to a Gaussian.
• Moment generating function — E[e^{tX}]; a transform whose derivatives at zero mint the moments (a light touch here; the ECE unit goes deeper).

The mental model

Expectation is a centre of mass: put the probability on a beam, and E[X] is where it balances. It need not be a possible value (2.4 children), and it is not the "typical" value when the distribution is skewed — incomes average far above the median because a long right tail drags the balance point. Linearity is the superpower: the expectation of a sum is the sum of expectations, even for entangled variables — which is why the expected value of complicated pipelines is often computable while their full distributions are hopeless, and why so many ML objectives are stated as expectations.

Variance is spread, in squared units — report the standard deviation when humans must read it. It is NOT linear: Var(X+Y) picks up a covariance cross-term, collapsing to the sum only under independence (an error bar of st2 depends on exactly this). Covariance's SIGN says together-or-opposite; its magnitude is unreadable until normalised into correlation — and correlation hears only LINEAR association: a perfect U-shaped dependence can score zero, a warning to plot before you summarise (st1).

Skewness and kurtosis are what the summary suppressed. Skew: one long tail — means dragged, medians honest, log-transforms often called for. Kurtosis: heavy tails — extremes far more common than the Gaussian bell predicts. Financial returns and real-world latencies are the classic heavy-tailed offenders: model them as Gaussian and you will price ruinous events as impossible. When a mean and variance are all you are shown, the professional reflex is to ask what the tails are doing.

The MGF, one polite paragraph: E[e^{tX}] packages every moment — differentiate at zero to mint them — and turns sums of independent variables into products. It is the machinery under the CLT's standard proof (pr5); recognise it, use it when a derivation asks, move on.

What you should now be able to explain or do

Compute E and Var for small discrete and continuous cases. Use linearity on a sum of dependent variables without flinching. Expand Var(X+Y) with its cross-term. Interpret a correlation and name its blind spot. Read skew and kurtosis warnings in real data.

Check yourself