1.15 Joint, marginal and conditional distributions

What this is and why it exists

One random variable is a warm-up; data is many at once — pixels, features, words — and the joint distribution is their complete law. This topic is the grammar of "several at once": marginalising out what you ignore, conditioning on what you learned, and the independence structures that make high-dimensional models possible at all. The multivariate normal is the worked example the whole of ML leans on.

The vocabulary

• Joint distribution — probability over combinations: P(X and Y together).
• Marginalisation — summing or integrating out the variables you are not asking about; the joint's shadow on one axis.
• Conditional distribution — the joint sliced at what you know, renormalised.
• Chain rule of probability — any joint factors as a cascade of conditionals: P(A)·P(B given A)·P(C given A,B)… — in any order.
• Independence — the joint factors into marginals; knowledge of one leaves the others untouched.
• Conditional independence — independent GIVEN a third thing; the workhorse assumption of graphical models and naive Bayes.
• Multivariate normal — the Gaussian in d dimensions: a mean vector and a covariance matrix, nothing else.

The mental model

Picture the joint of two variables as a hill over the plane. Marginalising is casting a shadow: flatten the hill onto one axis and you have that variable's distribution alone — what you get when you ignore the other. Conditioning is slicing: cut the hill at the observed value, rescale the slice to area one, and you have the other variable's law inside the world you learned. Shadow and slice are DIFFERENT curves in general, and their difference is dependence itself.

The chain rule says every joint is a story told one variable at a time: pick any order, multiply conditionals. Language models are literally this — the joint probability of a sentence factored as next-word-given-context, and "autoregressive" names the factorisation order. But raw conditionals on many variables are unlearnably many numbers; CONDITIONAL independence is the scissors that cut the story short: symptoms independent given the disease (naive Bayes), the future independent of the past given the present (Markov chains, the RL state). Two ordinary facts to hold apart: independent variables can become dependent once you condition on a common effect, and dependent ones can separate given a common cause — st5's collider and confounder, met here as pure probability.

The multivariate normal is the tractable paradise: specify a mean vector and a covariance matrix (symmetric PSD — la4's object, wearing its natural hat) and EVERYTHING follows. Its contours are ellipses whose axes are the covariance's eigenvectors (PCA's geometry again); every marginal is normal, every conditional is normal — with a mean formula that is exactly linear regression of one block on another; and zero covariance means full independence, a luxury only Gaussians grant (pr2 of the ECE unit says the same — same theorem). This closure under shadow and slice is why Gaussians power Kalman filters, Gaussian processes and every noise model with a tractable posterior.

What you should now be able to explain or do

Marginalise and condition a small discrete joint by hand. Factor a joint by the chain rule in two different orders. State conditional independence and give naive Bayes and Markov as instances. Describe the multivariate normal's closure properties and read its covariance geometrically.

Check yourself