Abstract: We propose smoothing spline mixed-effects density models for nonparametric estimations of density and conditional density functions with clustered data. The random effects in a density model introduce within-cluster correlation, enabling us to borrow strength across clusters by shrinking cluster-specific density functions to the population average, where the amount of shrinkage is decided by the data. Estimations are carried out using the penalized likelihood and are computed using a Markov chain Monte Carlo stochastic approximation algorithm. We derive an approximate cross-validation estimate of the aggregated Kullback-Leibler loss for the selection of the smoothing parameters. Our simulation study indicates that the proposed estimation method performs well. We apply our methods to investigate the evolution of hemoglobin density functions over time in response to guideline changes on anemia management for dialysis patients.

Key words and phrases: Markov chain Monte Carlo, penalized likelihood, random effects, smoothing spline ANOVA decomposition, stochastic approximation.