Solution
The Exact FDR of a Screen
Show the problem again
A screen tests 1000 hypotheses, of which exactly 800 are true nulls (with independent, exactly uniform p-values) and 200 are real effects. The p-values are mutually independent. The Benjamini-Hochberg procedure is run at nominal level q = 0.05. The false discovery rate is the expected value of (false rejections) / max(1, total rejections). What is the exact FDR of this procedure? Give a reduced fraction.
Worked solution
For independent p-values with uniform true nulls, the Benjamini-Hochberg procedure has FDR exactly (m0 / m) q, where m0 is the number of true nulls and m the number of tests. Here that is (800/1000) × 0.05 = 0.04 = 1/25. The result does not depend on how the 200 real-effect p-values are distributed, only on their independence from the null ones. The procedure is conservative by the factor m0/m because it is calibrated as if every null were true; a simulation of this setting agrees with 0.04 to within Monte Carlo error.
Source: Benjamini and Hochberg (1995) show that the BH procedure has FDR equal to pi0 q under independence. Statement written for AxiomIQ.