Abstract

Modern data analysis often involves challenges from distributed and

high-dimensional data, especially in fields such as finance and genomics. While

distributed statistical methods address fragmented datasets and subset selection

tackles high-dimensionality, existing approaches lack an effective combination of

these two aspects.

This paper proposes a communication-efficient distributed

algorithm for best-subset selection in generalized linear models. The method extends a state-of-the-art centralized best-subset selection approach to a distributed

framework, enabling scalable variable selection while optimizing computational

efficiency and communication cost. Theoretical guarantees are established, including statistical consistency and convergence rates.

Simulation studies and

real-world data analyses validate the method’s superior performance in terms

of predictive accuracy and computational efficiency, demonstrating its practical

relevance for modern data challenges.

Key words and phrases: Distributed statistical learning, Best-subset selection, Generalized linear models, Splicing technique

Information

Preprint No.SS-2025-0189
Manuscript IDSS-2025-0189
Complete AuthorsHongmei Lin, Xinxin Xiao, Chen Guo, Liu Yanlin, Tiejun Tong, Riquan Zhang
Corresponding AuthorsChen Guo
Emailsgc_1998@163.com

References

  1. Abramovich, F. & Grinshtein, V. (2016), ‘Model selection and minimax estimation in generalized linear models’, IEEE Transactions on Information Theory 62(6), 3721–3730.
  2. Akaike, H. (1998), Information theory and an extension of the maximum likelihood principle, in ‘Selected papers of Hirotugu Akaike’, Springer, pp. 199–213.
  3. Allouah, Y., Guerraoui, R., Gupta, N., Pinot, R. & Stephan, J. (2023), On the privacyrobustness-utility trilemma in distributed learning, in ‘International Conference on Machine Learning’, PMLR, pp. 569–626.
  4. Battey, H., Fan, J., Liu, H., Lu, J. & Zhu, Z. (2018), ‘Distributed testing and estimation under sparse high dimensional models’, The Annals of Statistics 46(3), 1352–1382.
  5. Beck, A. & Eldar, Y. C. (2013), ‘Sparsity constrained nonlinear optimization: Optimality conditions and algorithms’, SIAM Journal on Optimization 23(3), 1480–1509.
  6. Bertsimas, D., King, A. & Mazumder, R. (2016), ‘Best subset selection via a modern optimization lens’, The Annals of Statistics 44(2), 813 – 852.
  7. Blumensath, T. & Davies, M. E. (2009), ‘Iterative hard thresholding for compressed sensing’, Applied and Computational Harmonic Analysis 27(3), 265–274.
  8. Burnham, K. P. & Anderson, D. R. (2002), Model selection and multimodel inference: a practical information-theoretic approach, 2nd Edition, Springer.
  9. Chen, J. & Chen, Z. (2008), ‘Extended Bayesian information criteria for model selection with large model spaces’, Biometrika 95(3), 759–771.
  10. Chen, X., Liu, W. & Zhang, Y. (2019), ‘Quantile regression under memory constraint’, The Annals of Statistics 47(6), 3244–3273.
  11. Chen, X. & Xie, M.-G. (2014), ‘A split-and-conquer approach for analysis of extraordinarily large data’, Statistica Sinica 24(4), 1655–1684.
  12. Dedieu, A., Hazimeh, H. & Mazumder, R. (2021), ‘Learning sparse classifiers: Continuous and mixed integer optimization perspectives’, Journal of Machine Learning Research 22(135), 1–47.
  13. Fan, J., Guo, Y. & Wang, K. (2023), ‘Communication-efficient accurate statistical estimation’, Journal of the American Statistical Association 118(542), 1000–1010.
  14. Fan, J. & Li, R. (2001), ‘Variable selection via nonconcave penalized likelihood and its oracle properties’, Journal of the American Statistical Association 96(456), 1348–1360.
  15. Gao, Y., Liu, W., Wang, H., Wang, X., Yan, Y. & Zhang, R. (2022), ‘A review of distributed statistical inference’, Statistical Theory and Related Fields 6(2), 89–99.
  16. Hinterm¨uller, M., Ito, K. & Kunisch, K. (2002), ‘The primal-dual active set strategy as a semismooth Newton method’, SIAM Journal on Optimization 13(3), 865–888.
  17. Hocking, R. R. & Leslie, R. (1967), ‘Selection of the best subset in regression analysis’, Technometrics 9(4), 531–540.
  18. Jain, P., Tewari, A. & Kar, P. (2014), ‘On iterative hard thresholding methods for highdimensional m-estimation’, Advances in Neural Information Processing Systems 1, 685– 693.
  19. Jiang, R. & Yu, K. (2021), ‘Smoothing quantile regression for a distributed system’, Neurocomputing 466, 311–326.
  20. Jordan, M. I., Lee, J. D. & Yang, Y. (2019), ‘Communication-efficient distributed statistical inference’, Journal of the American Statistical Association 114(526), 668–681.
  21. Lee, K.-Y. & Courtade, T. (2020), Minimax bounds for generalized linear models, in H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan & H. Lin, eds, ‘Advances in Neural Information Processing Systems’, Vol. 33, Curran Associates, Inc., pp. 9372–9382.
  22. Li, M., Tian, Y., Feng, Y. & Yu, Y. (2024), ‘Federated transfer learning with differential privacy’, arXiv preprint arXiv:2403.11343 .
  23. Li, R., Lin, D. K. & Li, B. (2013), ‘Statistical inference in massive data sets’, Applied Stochastic Models in Business and Industry 29(5), 399–409.
  24. Li, T., Sahu, A. K., Talwalkar, A. & Smith, V. (2020), ‘Federated learning: Challenges, methods, and future directions’, IEEE Signal Processing Magazine 37(3), 50–60.
  25. Liu, K., Hu, S., Wu, S. Z. & Smith, V. (2022), ‘On privacy and personalization in cross-silo federated learning’, Advances in Neural Information Processing Systems 35, 5925–5940.
  26. Lozano, A., Swirszcz, G. & Abe, N. (2011), Group orthogonal matching pursuit for logistic regression, in G. Gordon, D. Dunson & M. Dud´ık, eds, ‘Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics’, Vol. 15 of Proceedings of Machine Learning Research, PMLR, Fort Lauderdale, FL, USA, pp. 452–460.
  27. Luo, J., Sun, Q. & Zhou, W.-X. (2022), ‘Distributed adaptive Huber regression’, Computational Statistics & Data Analysis 169, 107419.
  28. Lv, S. & Lian, H. (2022), ‘Debiased distributed learning for sparse partial linear models in high dimensions’, Journal of Machine Learning Research 23(2), 1–32.
  29. Mallat, S. G. & Zhang, Z. (1993), ‘Matching pursuits with time-frequency dictionaries’, IEEE Transactions on Signal Processing 41(12), 3397–3415.
  30. McCullagh, P. (2019), Generalized linear models, Routledge.
  31. Raskutti, G., Wainwright, M. J. & Yu, B. (2011), ‘Minimax rates of estimation for highdimensional linear regression over ℓq-balls’, IEEE Transactions on Information Theory 57(10), 6976–6994.
  32. Rockafellar, R. T. (1976), ‘Monotone operators and the proximal point algorithm’, SIAM Journal on Control and Optimization 14(5), 877–898.
  33. Rosenblatt, J. D. & Nadler, B. (2016), ‘On the optimality of averaging in distributed statistical learning’, Information and Inference: A Journal of the IMA 5(4), 379–404.
  34. Schwarz, G. (1978), ‘Estimating the dimension of a model’, The Annals of Statistics 6(2), 461– 464.
  35. Shamir, O., Srebro, N. & Zhang, T. (2014), Communication-efficient distributed optimization using an approximate Newton-type method, in ‘International Conference on Machine Learning’, PMLR, pp. 1000–1008.
  36. Shen, X., Pan, W. & Zhu, Y. (2012), ‘Likelihood-based selection and sharp parameter estimation’, Journal of the American Statistical Association 107(497), 223–232.
  37. Tibshirani, R. (1996), ‘Regression shrinkage and selection via the lasso’, Journal of the Royal Statistical Society Series B: Statstical Methodology 58(1), 267–288.
  38. Wang, L., Kim, Y. & Li, R. (2013), ‘Calibrating non-convex penalized regression in ultra-high dimension’, The Annals of Statistics 41(5), 2505–2536.
  39. Wang, X., Yang, Z., Chen, X. & Liu, W. (2019), ‘Distributed inference for linear support vector machine’, Journal of Machine Learning Research 20(113), 1–41.
  40. Wang, Y., Lu, W. & Lian, H. (2023), ‘Best subset selection for high-dimensional non-smooth models using iterative hard thresholding’, Information Sciences 625, 36–48.
  41. Zhang, C.-H. (2010), ‘Nearly unbiased variable selection under minimax concave penalty’, The Annals of Statistics 38(2), 894–942.
  42. Zhang, Y., Duchi, J. & Wainwright, M. (2015), ‘Divide and conquer kernel ridge regression: A distributed algorithm with minimax optimal rates’, Journal of Machine Learning Research 16(1), 3299–3340.
  43. Zhao, Z. & Lian, H. (2025), ‘Distributed estimation for ℓ0-constrained quantile regression using iterative hard thresholding’, Mathematics 13(4), 669.
  44. Zhu, J., Wen, C., Zhu, J., Zhang, H. & Wang, X. (2020), ‘A polynomial algorithm for best-subset selection problem’, Proceedings of the National Academy of Sciences 117(52), 33117–33123.
  45. Zou, H. & Hastie, T. (2005), ‘Regularization and variable selection via the elastic net’, Journal of the Royal Statistical Society Series B: Statistical Methodology 67(2), 301–320. Hongmei Lin, Shanghai University of International Business and Economics, Shanghai, China

Acknowledgments

We sincerely thank the editor, the associate editor and two referees for

their valuable comments and constructive suggestions. Hongmei Lin’s research was supported in part by the Shanghai Oriental Talent Program

(Youth Project) and the National Natural Science Foundation of China

(12171310). Tiejun Tong’s research was supported in part by the General

Research Fund of Hong Kong (HKBU12300123) and the Initiation Grant

for Faculty Niche Research Areas of Hong Kong Baptist University (RC-

FNRA-IG/23-24/SCI/03). Riquan Zhang’s research was supported in part

by the National Nature Science Foundation of China (12371272, 12531013).

Supplementary Materials

The supplementary materials contain the technical conditions required for

the theorems, the complete proofs of all lemmas and theorems, and the

additional numerical results.

Supplementary materials are available for download.