1.20 Sampling, standard error and confidence intervals

What this is and why it exists

Every number computed from a sample would have come out differently with a different sample — so an honest number carries an error bar. This topic builds the error bar from first principles (sampling distribution, standard error, confidence interval), prices it (root n), and hands you the modern tool for when formulas run out: the bootstrap.

The vocabulary

• Population vs sample — the whole you care about versus the part you measured.
• Sampling bias — a sample that systematically misrepresents its population; no size cures it.
• Sampling distribution — how a statistic varies across imagined repeated samples (pr5's object, now put to work).
• Standard error (SE) — that distribution's standard deviation; for a mean, s over root n.
• Confidence interval (CI) — statistic ± about two SEs (at 95%): the range of plausible true values given the sampling noise.
• Bootstrap — resample your own sample, with replacement, thousands of times; the spread of recomputed statistics estimates the sampling distribution empirically.

The mental model

Imagine the study repeated a thousand times: a thousand samples, a thousand slightly different means. That cloud of means is the sampling distribution, and its width — the standard error — is how much your one mean should be trusted. The CLT (pr5) supplies the machinery: the cloud is Gaussian with width s over root n, so mean ± two SEs brackets the truth in about 95% of repetitions. That is a confidence interval, assembled from parts you already own.

Say carefully what it does and does not mean. The PROCEDURE captures the truth 95% of the time — that is a property of the method across repetitions, not a 95% probability that the truth sits in this one interval (the truth is a fixed number, not a random one; the Bayesian credible interval of st6 is the object that makes the tempting sentence true). What a CI is FOR, daily: a decision aid — an interval spanning zero means the sign of the effect is not settled; a wide interval means the data cannot answer the question yet at this n.

Two budget clauses. Root-n pricing: SE shrinks with the square root, so each halving of the interval costs quadruple the data (pr5's law, now in rupees). And bias is not bought off by n: a survey of app users describes app users perfectly at any size — sampling MORE of the wrong population narrows the interval around the wrong answer. Ask "who could never appear in this sample?" before admiring any interval; a biased big sample is confidence in error.

The bootstrap is the general-purpose escape when no SE formula exists — a median, a correlation, a model's AUC. Treat the sample as its own population: resample n rows WITH replacement, recompute the statistic, repeat a few thousand times, and read the spread of results as the sampling distribution — percentiles give the interval. It works because the sample's shape is the best available stand-in for the population's; it fails where that stand-in fails: tiny samples, extreme tails, dependent rows (resample BLOCKS for time series). One loop, honest error bars for almost anything — reason enough for this topic to exist.

What you should now be able to explain or do

Compute an SE and a 95% CI for a mean. State the CI's meaning without the classic misreading. Apply root-n budgeting to a precision request. Detect a sampling-bias story that size cannot fix. Bootstrap a CI for a statistic with no formula.

Check yourself