Abstract: We investigate the problem of deciding on the number of hypotheses to be considered in a multiple hypothesis testing framework when the overall number of observations that can be collected is limited. A natural question in this context is whether the number of hypotheses to be tested should be limited in favor of additional observations per considered hypothesis. We provide guidelines concerning the choice of an optimum number of considered hypotheses in common testing situations. The optimization is with respect to the expected number of correct rejections in the hypothesis testing context. We also briefly discuss the classification setting, where a linear combination of true and false positives is considered. The overall number of observations may be limited for several reasons, such as the number of patients or the amount of probe material available. We demonstrate that considering an appropriate number of hypotheses in this context can lead to a substantial increase in the expected number of correct rejections.
Key words and phrases: Bonferroni rule, classification, Dunnett test, false discovery rate, multiple hypotheses testing.