Abstract: Subgroup analysis with unspecified subgroup memberships has received increasing attention in recent years. In Shen and He (2015), a structured logisticnormal mixture model was proposed to characterize the subgroup distributions and the subgroup membership simultaneously, but under the assumption that the subgroups differ only in the means. In this paper, we consider a penalized likelihood approach for more general cases with heterogeneous subgroup variances. Despite substantial technical complications in the development of the statistical theory, we show that the penalized likelihood inference for the existence of subgroups and for the estimation of subgroup membership can be carried out in the existing framework. Empirical results with a simulation study and two data examples demonstrate the usefulness of the proposed method.

Key words and phrases: EM algorithm; heterogeneous components; homogeneity test; likelihood ratio test; mixture models; subgroup identification.