Abstract: Let X₁,…,X_n be independent observations with X_i ~ N(θ_i,1), where (θ₁,…,θ_n) is an unknown vector of normal means. Let f_n(x) = ∑ _i=1ⁿ(d∕dx)P_n{X_i ≤ x}∕n be the average marginal density of observations. We consider the problem of testing H₀:f_n ∈₀, where ₀ is a family of mixture densities. This includes detecting nonzero normal means with ₀ = {f_δ0} and testing homogeneity in mixture models with ₀ = {f_δμ}. We study a generalized likelihood ratio test (GLRT) based on the generalized maximum likelihood estimator (GMLE, Robbins (1950); Kiefer and Wolfowitz (1956)). We establish a large deviation inequality that provides a divergence rate _n of the GLRT under the null hypothesis. The inequality implies that the significance level of the test is of equal or smaller order than n_n². We show that the test can detect any alternative that is separated from the null by Hellinger distance _n. For the two-component Gaussian mixture, it turns out that the GLRT has full power asymptotically throughout the same region of amplitude sparsity where the Neyman-Pearson likelihood ratio test separates the two hypotheses completely (Donoho and Jin (2004)). We demonstrate the power of the GLRT for moderate samples with numerical experiments.

Key words and phrases: Detection boundary, generalized likelihood ratio test, generalized maximum likelihood estimator, normal mixture, sparse normal means.