Abstract: In this paper, we propose a new clustering approach for multivariate responses in a longitudinal analysis. Clustering analyses for multiple outcomes can be challenging, owing to multiple sources of correlation from multiple outcomes of the same subject and longitudinal measurements. The proposed method enhances clustering analyses by integrating multiple sources of correlations. Specifically, we incorporate random effects to capture correlations from multivariate responses, and group individuals by penalizing the pairwise distances between the B-spline coefficient vectors. We implement an alternating directions and method of multipliers (ADMM) algorithm for optimization in clustering. Furthermore, we study the asymptotic convergence rate of the proposed nonparametric estimator in the presence of longitudinal correlations for the random-effects model. The results of simulations and a real-data analysis show that the proposed method outperforms existing clustering methods.

Key words and phrases: ADMM, minimax concave penalty, model selection, penalized-spline, random effects.