Solution
The Smallest of Ten p-values
Show the problem again
Ten independent hypothesis tests are run, and in fact all ten null hypotheses are true, so each p-value is uniform on [0, 1]. The smallest of the ten p-values is reported. If the reported value is 0.004, what is the probability, under this all-null setting, that the smallest of ten p-values would be at most 0.004? (This is the Sidak-adjusted p-value.) Give your answer rounded to 4 decimal places; 3 decimal places is also accepted.
Worked solution
The minimum is at most 0.004 unless all ten p-values exceed 0.004. By independence this happens with probability 0.996^10, so the required probability is 1 - 0.996^10 ≈ 0.0393. The naive reading "p = 0.004, so strong evidence" ignores that the reported value is the best of ten tries; the Bonferroni bound 10 × 0.004 = 0.04 is slightly larger than the exact value 0.0393 because it ignores overlap between the events.
Source: Standard Sidak-correction exercise on selecting the best of several tests. Statement written for AxiomIQ.