The Peeking Problem
An analyst runs an A/B test planned for 30 days, but checks the p-value every day and stops to declare significance the first time p < 0.05. Does this procedure preserve the advertised 5% false-positive rate? Answer yes or no.
Related problems & prerequisites
Worth solving first
- What a p-value Means 6.0/10
- Simpson's Paradox 6.0/10
- Halving the Standard Error 5.0/10
Source: Optional-stopping bias, a standard cautionary result in the statistical-inference literature. Statement written for AxiomIQ.