Solution
Two Hits Among Twenty Nulls
Show the problem again
A researcher tests 20 independent hypotheses, all of which are true nulls, each at significance level 0.05 (so each p-value is uniform on [0, 1]). What is the probability that at least two of the tests come out significant? Give your answer rounded to 4 decimal places; 3 decimal places is also accepted.
Worked solution
The number of significant results is Binomial(20, 0.05). P(0) = 0.95^20 ≈ 0.3585 and P(1) = 20 × 0.05 × 0.95^19 ≈ 0.3774, so P(at least 2) = 1 - 0.3585 - 0.3774 ≈ 0.2642. More than one run in four of an entirely empty study yields two or more "findings", which is why requiring replication or a multiplicity correction matters.
Source: Standard multiple-comparisons illustration of chance findings. Statement written for AxiomIQ.