The Peeking Problem
An analyst runs an A/B test planned for 30 days, but checks the p-value every day and stops to declare significance the first time p < 0.05. Does this procedure preserve the advertised 5% false-positive rate? Answer yes or no.
Related problems & prerequisites
Worth solving first
Source: Optional-stopping bias, a standard cautionary result in the statistical-inference literature. Statement written for AxiomIQ.