1.22 A/B testing and experiment design

What this is and why it exists

An A/B test is the one place most practitioners personally run the scientific method — and the place where every sin of st3 gets committed with money on the line. This topic is experiment design as craft: randomise so the comparison means something, size the test BEFORE starting, refuse to peek, and read a readout like a referee rather than a fan.

The vocabulary

• Randomization — assigning users to variants by coin flip, severing every link between who-they-are and what-they-saw.
• Control — the current experience, running alongside as the baseline.
• Blocking / stratification — balancing known heavy-hitter variables (platform, country, new-vs-returning) across variants by design rather than luck.
• Minimum detectable effect (MDE) — the smallest lift worth acting on; the input that sizes the test.
• Sample-size calculation — from MDE, baseline rate, α and power: how many users, hence how many weeks — computed BEFORE launch.
• Peeking — checking significance repeatedly and stopping on the first happy result; the classic self-deception.
• Sequential testing — methods built to be monitored continuously, paying for the privilege with wider bounds.
• Novelty effect — early lift caused by newness itself, decaying as users habituate.

The mental model

Randomization is what makes the comparison causal — st5 will say it formally, but the intuition fits here: coin-flip assignment breaks every possible link between user traits and variant, so a persisting difference has one remaining explanation. Corollary with teeth: any post-launch difference in WHO is in each bucket (more mobile users in B) is a bug in the pipeline, not a finding in the data — validate bucket balance before believing anything downstream. Blocking is insurance for the traits you know matter: stratify assignment on platform or geography so balance is guaranteed, not merely probable.

Sizing comes before launching, and it is a negotiation, not a formality. The MDE is a BUSINESS question — what lift would actually change a decision? — and with it, the baseline rate, α and target power (0.8 conventionally), the calculator returns the required n per variant, which divided by traffic is the test's duration in weeks. Run the arithmetic before launch and you sometimes learn the humbling truth early: the traffic cannot detect the effect that matters in acceptable time — better to know on Monday than after six wasted weeks. This calculation is also the contract that makes stopping honest: the end date is fixed BEFORE the first user arrives.

Peeking is how honest people cheat. Watching the dashboard daily and stopping the first time p dips under 0.05 turns a 5% false-alarm budget into 30% or worse — each look is another lottery ticket (st3's multiple comparisons, spread over time). The rule: no significance decisions before the pre-computed end. When business genuinely demands early stopping, use sequential methods DESIGNED for continuous monitoring — they keep the error budget by demanding stronger evidence at each look. And two calendar effects to respect: novelty (a redesign's early lift decays as the newness wears off — run past it, comparing early and late weeks) and weekly rhythm (weekend users differ from weekday users — run whole weeks, never four days).

Reading a readout critically is the transferable skill: Was assignment randomised, and do the buckets balance? Was n pre-computed, and did the test run its planned course — or stop early on a good day? Is the metric the PRIMARY one chosen in advance, or the best of twenty (st3's hacking)? Is the effect size with its interval reported, or only a p? Does the lift persist past novelty? Five questions; most weak readouts fail by the third.

What you should now be able to explain or do

Design an A/B test end to end: hypothesis, primary metric, MDE, sample size, duration. Explain what randomization buys and what bucket imbalance signals. Say why peeking inflates false alarms and what sequential testing changes. Interrogate a readout with the five questions.

Check yourself