Solution
The Peeking Problem
Show the problem again
An analyst runs an A/B test planned for 30 days, but checks the p-value every day and stops to declare significance the first time p < 0.05. Does this procedure preserve the advertised 5% false-positive rate? Answer yes or no.
Worked solution
No, optional stopping inflates the Type I error rate far above 5%. The 5% guarantee applies to a single look at a fixed sample size. Repeatedly testing gives the noise many chances to cross the threshold; with daily peeking over a month, the false positive rate can exceed 25–30%, since under the null the p-value process will often dip below 0.05 at some point even with no real effect. Fixes include pre-registered sample sizes, alpha-spending/group-sequential boundaries, or always-valid sequential methods.
Source: Optional-stopping bias, a standard cautionary result in the statistical-inference literature. Statement written for AxiomIQ.