Solution
Trading Bias for Variance
Show the problem again
Six independent observations are drawn from a normal distribution with unknown mean and variance σ². Let S = Σ(Xᵢ − X̄)² be the sum of squared deviations about the sample mean, and consider estimators of σ² of the form S/d for a constant d > 0. Choose d to minimise the mean squared error E[(S/d − σ²)²] (the same d works for every σ²). Let σ = 1 for definiteness: what is the minimum mean squared error? Give a reduced fraction.
Worked solution
S/σ² has a chi-square distribution with n − 1 = 5 degrees of freedom, so E[S] = 5σ² and Var(S) = 10σ⁴. For the estimator S/d, MSE = Var + bias² = [10 + (5 − d)²]σ⁴/d². Differentiating, the minimiser is d = (n − 1) + 2 = n + 1 = 7, the same for every σ. At d = 7 the MSE is (10 + 4)/49 = 2/7 σ⁴. For comparison, the unbiased choice d = 5 gives 2/5 and the maximum-likelihood choice d = 6 gives 11/36, both worse than 2/7. The unbiased estimator is not the best in the mean-squared-error sense: a little shrinkage towards zero pays for itself. The answer is 2/7.
Source: Original problem written for AxiomIQ.