Statistica Sinica 31 (2021), 959-979
Sara Algeri1,2 and David A. van Dyk2
Abstract: In applied settings, hypothesis testing when a nuisance parameter is identifiable only under the alternative often reduces to a problem of testing one hypothesis multiple times (TOHM). Specifically, a fine discretization of the space of the nonidentifiable parameter is specified, and the null hypothesis is tested against a set of sub-alternative hypotheses, one for each point of the discretization. The resulting sub-test statistics are then combined to obtain a global p-value. We propose a computationally efficient inferential tool to perform TOHM under stringent significance requirements, such as those typically required in the physical sciences, (e.g., a p-value < 10− 7). The resulting procedure leads to a generalized approach to performing inferences under nonstandard conditions, including non-nested model comparisons.
Key words and phrases: Bump hunting, multiple hypothesis testing, non-identifiability in hypothesis testing, non-nested models comparison.