Solution
The Average p-value of a True Null
Show the problem again
A fair coin is flipped 4 times and the number of heads $X$ is recorded. The one-sided exact p-value for testing $H_0:p=1/2$ against $H_1:p>1/2$ is $p(X)=P(\text{Binomial}(4,1/2)\ge X)$. When $H_0$ is true, what is the expected value $E[p(X)]$? Give a reduced fraction.
Worked solution
Tail probabilities $P(\ge x)$ in sixteenths for $x=0..4$ are $16,15,11,5,1$, and $X$ has weights $1,4,6,4,1$ out of 16. Hence $E[p(X)]=\dfrac{1\cdot16+4\cdot15+6\cdot11+4\cdot5+1\cdot1}{16\cdot16}=\dfrac{163}{256}\approx0.637.$ A continuous test statistic would give $1/2$; the discreteness makes the p-value stochastically larger than uniform (conservative), and the average is noticeably above one half.
Source: Original problem written for AxiomIQ. Related to the known fact that discrete p-values are stochastically larger than uniform.