1.21 Hypothesis testing and p-values

What this is and why it exists

A difference showed up — between two groups, two models, two months. Hypothesis testing asks the deflating question first: could chance alone have produced this? This topic teaches the ritual honestly: what a p-value is and famously is not, which named test fits which situation, and the two ways the ritual is gamed — because you will referee gamed versions of it for the rest of your career.

The vocabulary

• Null hypothesis — the boring explanation: no effect, no difference; the thing the test tries to embarrass.
• Alternative hypothesis — some effect exists.
• p-value — the probability of data at least this extreme IF the null were true. Not the probability the null is true.
• Significance level (α) — the false-alarm rate you accept in advance, conventionally 0.05.
• Type I / Type II error — false alarm / missed effect; their rates trade off.
• Power — the probability of catching a real effect of a given size; the enemy of Type II.
• Effect size — HOW BIG the difference is, in units someone cares about — the number significance forgets.
• t-test / chi-square / ANOVA — the standard instruments: comparing means; testing counts against expectations or independence; comparing three or more group means at once.

The mental model

A test is a courtroom where chance is the defendant. Assume the null — pure chance — and ask how surprising the observed data would be in that world. The p-value is that surprise, and a small one says "chance is a strained explanation"; below the pre-agreed α, the null is rejected. Every correct reading of a p-value walks through the null's world: it is P(data this extreme GIVEN no effect) — and flipping it into P(no effect given data) is the exact base-rate mistake of pr1, committed in a lab coat.

Picking the instrument is mostly pattern-matching: two group MEANS on a numeric outcome — t-test (checking st1's shape first; heavy tails and tiny n want care). COUNTS in categories against an expectation, or two categorical variables' independence — chi-square. THREE OR MORE group means — ANOVA, whose F-statistic asks whether between-group differences exceed within-group noise; run it as one test rather than testing every pair, for reasons two sentences away.

The two honesty clauses carry the topic. First: significance is not size. With n huge, a 0.2% conversion difference achieves p under 0.001 — real, and possibly worthless; with n tiny, a large real effect fails to reach significance — absence of evidence, not evidence of absence. Always report the effect size with its interval (st2); "significant" answers "is it chance?", never "does it matter?". Second: the ritual is gameable, and the games have names. P-HACKING: test twenty outcomes and one will clear 0.05 by luck — twenty lottery tickets, not one finding; the MULTIPLE-COMPARISONS corrections (Bonferroni's blunt division, or false-discovery-rate control) re-price the tickets honestly. PEEKING: testing repeatedly as data accumulates and stopping at the first significant moment inflates false alarms far beyond α (st4 meets this as its main villain). The referee's questions, always: how many things were tested, when was the stopping rule fixed, and where is the effect size?

What you should now be able to explain or do

State a p-value's definition without the flip. Choose t-test, chi-square or ANOVA from a scenario. Explain the power–α trade and what raises power. Distinguish significant from important with an n-based example each way. Name p-hacking and peeking and the corrections.

Check yourself