S4-5.2 Operations on Random Variables

What this is and why it exists

A full density function is more information than most engineering questions need. Moments compress it: the mean says where, the variance says how spread, the higher moments describe the shape's lean and tails. This unit builds the compression toolkit, the inequalities that turn two moments into guarantees when the distribution is unknown, the transform tools that crack sums of random variables, and the multivariate case that dominates practice — jointly Gaussian variables, the one family where correlation is the whole story.

The vocabulary

• Expectation — the probability-weighted average of a random variable or any function of it.
• Moments about the origin / central moments — averages of powers of X, and of powers of X minus its mean; variance is the second central moment.
• Markov's inequality — for a non-negative variable, the probability of exceeding a exceeds nothing more than mean over a.
• Chebyshev's inequality — the probability of straying k standard deviations from the mean is at most one over k squared, for ANY distribution.
• Characteristic function — the expectation of e to the jωX: the density's Fourier transform, existing always.
• Moment-generating function — the expectation of e to the tX: same purpose, real exponent, when it exists.
• Central Limit Theorem (CLT) — properly scaled sums of many independent variables converge to Gaussian.
• Jointly Gaussian — a vector of variables every linear combination of which is Gaussian; described completely by means and a covariance matrix.

The mental model

Moments are a compression codec for distributions. Two numbers — mean and variance — are a lossy summary, and the inequalities state exactly what the summary still guarantees. Markov: crude, needing only non-negativity and the mean. Chebyshev: sharper, spending the variance to bound both tails at once — at most 1/9 of any distribution lies beyond three standard deviations, no matter how pathological. These bounds are loose by design; their power is universality, and they are the ancestors of every concentration argument used in ML theory.

The transform tools exist for one dominant reason: sums. The density of a sum of independent variables is a convolution — painful — but transforms turn convolution into multiplication: the characteristic function of X plus Y is the product of the individual ones. Moments also fall out by differentiation at zero, which names the MGF. And the CLT becomes visible from here: multiply n identical characteristic functions, scale properly, and the product converges to the Gaussian's — which is WHY summed randomness is Gaussian, and why the Gaussian owns noise modelling: every macroscopic noise source is a sum of microscopic ones.

Jointly Gaussian vectors close the unit as the multivariate royalty. The definition is stronger than "each component Gaussian": every linear combination must be Gaussian. The reward is total description by first and second moments — means, variances, correlations — and two properties no other family offers so cleanly: uncorrelated jointly Gaussian variables are fully independent (elsewhere uncorrelated is weaker), and any linear processing of a Gaussian vector yields another Gaussian vector. Linear filters process Gaussian noise into Gaussian noise; the whole of linear receiver analysis leans on that closure.

What you should now be able to explain or do

Compute means, variances and central moments from a density. Apply Markov and Chebyshev and state what each requires. Use a characteristic or moment-generating function to find moments and to add independent variables. State the CLT precisely and the two special properties of jointly Gaussian vectors.

Check yourself