Abstract: Estimating finite mixture models is a fundamental and challenging problem. We propose a penalized method for a Gaussian mixture linear regression, where the error terms follow a location–scale mixture of Gaussian distributions. The objective function is a combination of the likelihood function of the observed data and a penalty on the pairwise differences of the parameters. We develop an alternating direction method of multipliers algorithm, and establish its convergence property. By clustering and merging similar observations in an automatic manner, our method provides an integrated tool for simultaneously determining the number of components and estimating the parameters in finite mixture models. Moreover, the proposed method allows the mean and precision parameters to have different structures, enabling us to obtain pooled estimators. We also establish the statistical properties of our estimators. Extensive simulations and real-data examples are presented to evaluate the numerical performance of the proposed method.

Key words and phrases: Alternating direction method of multipliers, consistency, linear regression, pairwise difference, pooled estimator.