1.12 Probability foundations and Bayes

What this is and why it exists

Machine learning is belief-updating at industrial scale, and Bayes' theorem is the law that says how updating is done correctly. Humans do it wrongly by instinct — the base-rate trap fools doctors and judges — which is exactly why the two worked examples here (the medical test, the spam filter) are the module's most transferable twenty minutes.

The vocabulary

• Sample space and events — everything that could happen, and the subsets we assign probability to.
• The axioms — probabilities are non-negative, total one, and add across mutually exclusive events; everything else is derived.
• Conditional probability — P(A given B): the probability of A inside the world where B happened.
• Independence — B's occurrence tells you nothing about A: the joint probability factors.
• Total probability — P(A) assembled from its probability under each cause, weighted by the causes' probabilities.
• Bayes' theorem — P(cause given evidence) from P(evidence given cause), the prior, and the evidence's total probability.
• Prior, likelihood, posterior — belief before, the evidence's voice, belief after.

The mental model

Conditioning is shrinking the universe. P(A given B) throws away every world where B did not happen and renormalises what is left — the same slice-and-rescale picture the ECE probability unit uses, because it is the same mathematics. Independence is when the slicing changes nothing.

Bayes is the direction-reverser. What you can measure runs forward — P(positive test GIVEN disease), P(these words GIVEN spam) — and what you need runs backward: P(disease GIVEN positive). The theorem swaps them at the price of two more ingredients: the PRIOR (how common the cause is before evidence) and the evidence's total probability (the forward probability under every cause, weighted — total probability doing its one job).

Now the trap, with numbers. A disease afflicts 1 in 1000; a test catches 99% of cases and false-alarms on 2% of the healthy. A random person tests positive — how worried should they be? Instinct says 99%. Count instead: in 100,000 people, about 100 have the disease and ~99 test positive; 99,900 are healthy and ~1998 test positive anyway. Positives: 2097, of which 99 are real — under 5%. The prior (1 in 1000) dominates the accuracy (99%), and IGNORING the prior is the base-rate trap. The spam filter is the same computation run on purpose: words common in spam raise the posterior, the prior is the overall spam rate, and "naive" Bayes (classical-ML module) is this with a cheeky independence assumption across words.

Every detector — medical, fraud, defect, intrusion — obeys this arithmetic: when the condition is rare, even excellent tests produce mostly false alarms. That single sentence, believed in the bones, is worth the whole topic.

What you should now be able to explain or do

Derive Bayes from the definition of conditional probability. State independence three ways. Run the medical-test computation with fresh numbers and locate the trap. Explain a spam filter as posterior computation.

Check yourself