Abstract

We study high-dimensional binary classification across multiple related but heterogeneous

tasks.

For each task, the proposed method models the task-specific coefficient vector of a linear

support vector machine (SVM) as the sum of a sparse common component and a row-sparse taskspecific deviation, thereby separating features shared across tasks from those that capture cross-task

heterogeneity. To handle the nonsmooth hinge loss, we employ a convolution-smoothing technique and

estimate the common and deviation components through a composite regularizer that combines an

ℓ1 penalty with a row-wise ℓ2,1 penalty. Theoretically, we establish nonasymptotic estimation error

bounds under high-dimensional settings, with the rate explicitly decomposing the stochastic error from

the smoothing bias. We further prove variable selection consistency for both the shared active set and

the heterogeneous active row set under irrepresentability and beta-min conditions. For computation,

we develop a projected proximal-gradient algorithm tailored to the constrained composite optimization

problem. Simulation studies and an application to PISA data illustrate the finite-sample performance

of the proposed method relative to pooled and task-specific alternatives.

Key words and phrases: Convolution-smoothing, heterogeneous data, high-dimensional classification, multitask learning, support vector machine, variable selection

Information

Preprint No.SS-2026-0338
Manuscript IDSS-2026-0338
Complete AuthorsLing Peng, Yue Wang, Xiaohui Liu, Heng Lian
Corresponding AuthorsYue Wang
Emailsyuewangcityu@163.com

References

  1. Ando, R. K. and Zhang, T. “A framework for learning predictive structures from multiple tasks and unlabeled data.”
  2. Journal of Machine Learning Research, 6(61):1817–1853 (2005).
  3. Argyriou, A., Evgeniou, T., and Pontil, M. “Convex multi-task feature learning.” Machine Learning, 73(3):243–272
  4. (2008).
  5. Bickel, P. J., Ritov, Y., and Tsybakov, A. B. “Simultaneous analysis of Lasso and Dantzig selector.” The Annals of
  6. Statistics, 37(4):1705–1732 (2009).
  7. Cai, T., Liu, M., and Xia, Y. “Individual data protected integrative regression analysis of high-dimensional heterogeneous data.” Journal of the American Statistical Association, 117(540):2105–2119 (2022).
  8. Chen, J., Liu, J., and Ye, J. “Learning incoherent sparse and low-rank patterns from multiple tasks.” In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1179–1188.
  9. ACM (2010).
  10. Chen, J., Zhou, J., and Ye, J. “Integrating low-rank and group-sparse structures for robust multi-task learning.”
  11. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,
  12. 42–50. ACM (2011).
  13. Cortes, C. and Vapnik, V. “Support-vector networks.” Machine Learning, 20(3):273–297 (1995).
  14. Cristianini, N. and Shawe-Taylor, J. An Introduction to Support Vector Machines and Other Kernel-Based Learning
  15. Methods. Cambridge: Cambridge University Press (2000).
  16. Duan, R., Ning, Y., and Chen, Y. “Heterogeneity-aware and communication-efficient distributed statistical inference.”
  17. Biometrika, 109(1):67–83 (2022).
  18. Evgeniou, T., Micchelli, C. A., and Pontil, M. “Learning multiple tasks with kernel methods.” Journal of Machine
  19. Learning Research, 6(21):615–637 (2005).
  20. Evgeniou, T. and Pontil, M. “Regularized multi-task learning.” In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 109–117. ACM (2004).
  21. Fernandes, M., Guerre, E., and Horta, E.
  22. “Smoothing quantile regressions.” Journal of Business & Economic
  23. Statistics, 39(1):338–357 (2021).
  24. Horowitz, J. L. “A smoothed maximum score estimator for the binary response model.” Econometrica, 60(3):505–531
  25. (1992).
  26. Jacob, L., Vert, J.-P., and Bach, F. R. “Clustered multi-task learning: A convex formulation.” In Advances in Neural
  27. Information Processing Systems 21, 745–752. Curran Associates, Inc. (2008).
  28. Koo, J.-Y., Lee, Y., Kim, Y., and Park, C. “A Bahadur representation of the linear support vector machine.” Journal
  29. of Machine Learning Research, 9(44):1343–1368 (2008). minimax optimality.” Journal of the Royal Statistical Society: Series B (Statistical Methodology), 84(1):149–173
  30. (2022).
  31. Lian, H. and Fan, Z. “Divide-and-conquer for debiased L1-norm support vector machine in ultra-high dimensions.”
  32. Journal of Machine Learning Research, 18(182):1–26 (2018).
  33. Meier, L., van de Geer, S., and B¨uhlmann, P. “The group lasso for logistic regression.” Journal of the Royal Statistical
  34. Society: Series B (Statistical Methodology), 70(1):53–71 (2008).
  35. Negahban, S. N., Ravikumar, P., Wainwright, M. J., and Yu, B. “A unified framework for high-dimensional analysis
  36. of M-estimators with decomposable regularizers.” Statistical Science, 27(4):538–557 (2012).
  37. Obozinski, G., Wainwright, M. J., and Jordan, M. I.
  38. “Support union recovery in high-dimensional multivariate
  39. regression.” The Annals of Statistics, 39(1):1–47 (2011).
  40. Park, C., Kim, K.-R., Myung, R., and Koo, J.-Y. “Oracle properties of SCAD-penalized support vector machine.”
  41. Journal of Statistical Planning and Inference, 142(8):2257–2270 (2012).
  42. Peng, B., Wang, L., and Wu, Y. “An error bound for L1-norm support vector machine coefficients in ultra-high
  43. dimension.” Journal of Machine Learning Research, 17(233):1–26 (2016).
  44. Rybak, J., Battey, H., and Zhou, W.-X. “On inference for the support vector machine.” Journal of Machine Learning
  45. Research, 26(85):1–54 (2025). Sch¨olkopf, B. and Smola, A. J. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and
  46. Beyond. Cambridge, MA: MIT Press (2002).
  47. Steinwart, I. and Christmann, A. Support Vector Machines. Information Science and Statistics. New York: Springer
  48. (2008).
  49. Tan, K. M., Wang, L., and Zhou, W.-X. “High-dimensional quantile regression: Convolution smoothing and concave regularization.”
  50. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 84(1):205–233
  51. (2022)
  52. Tibshirani, R., Saunders, M., Rosset, S., Zhu, J., and Knight, K. “Sparsity and smoothness via the fused lasso.”
  53. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(1):91–108 (2005).
  54. Vapnik, V. N. The Nature of Statistical Learning Theory. Information Science and Statistics. New York: Springer,
  55. 2 edition (2000).
  56. Wainwright, M. J. High-Dimensional Statistics: A Non-Asymptotic Viewpoint. Cambridge Series in Statistical and
  57. Probabilistic Mathematics. Cambridge: Cambridge University Press (2019).
  58. Wang, B., Zhou, L., Gu, Y., and Zou, H. “Density-convoluted support vector machines for high-dimensional classification.” IEEE Transactions on Information Theory, 69(4):2523–2536 (2023).
  59. Wang, B., Zhou, L., Yang, J., and Mai, Q. “Density-convoluted tensor support vector machines.” Statistics and Its
  60. Interface, 17(2):231–247 (2024).
  61. Wang, L., Zhu, J., and Zou, H. “The doubly regularized support vector machine.” Statistica Sinica, 16(2):589–615
  62. (2006).
  63. Wang, Y. and Lian, H. “Sparse additive support vector machines in bounded variation space.” Information and
  64. Inference: A Journal of the IMA, 13(1):iaae003 (2024).
  65. Wang, Y., Lian, H., and Zhou, W. “Gbs-SVM: A linearly convergent gossip algorithm for high-dimensional classification.” Available at SSRN 6270798 (2026).
  66. Wolfson, M., Wallace, S. E., Masca, N., Rowe, G., Sheehan, N. A., Ferretti, V., LaFlamme, P., Tobin, M. D.,
  67. Macleod, J., Little, J., Fortier, I., Knoppers, B. M., and Burton, P. R. “DataSHIELD: Resolving a conflict in contemporary bioscience—performing a pooled analysis of individual-level data without sharing the data.”
  68. International Journal of Epidemiology, 39(5):1372–1382 (2010).
  69. Yuan, M. and Lin, Y. “Model selection and estimation in regression with grouped variables.” Journal of the Royal
  70. Statistical Society: Series B (Statistical Methodology), 68(1):49–67 (2006). Zh Y d Y Q “A lti t k l i ” IEEE T ti K l d d D t E i i
  71. 34(12):5586–5609 (2022). Ling Peng

Acknowledgments

The authors thank the Editor, the Associate Editor, and two anonymous reviewers for their

constructive comments and suggestions, which have greatly improved the paper. The research of Ling Peng is supported by Jiangxi Provincial Natural Science Foundation 20252BAC240162

and the China Postdoctoral Science Foundation 2024M751232.

Yue Wang’s research is

supported by National Natural Science Foundation of China (Grant No.12501416), Anhui

Provincial Natural Science Foundation (Grant No.2508085QA002), and the Fundamental

Research Funds for the Central Universities (Grant No.WK2040250127). Xiaohui Liu’s research is supported by the NSF of China (Grant No.12471257) and the NSF of the Science

and Technology Department of Jiangxi Province (Grant No. 20243BCE51010), the NSSF of

China (Grant No. 24&ZD069). This work is also supported by the Jiangxi Province Key

Laboratory of Data Science in Finance and Economics (Grant No. 2024SSY03201).

Supplementary Materials

The Supplementary Material contains specific examples of smoothed hinge loss, detailed

algorithmic descriptions, and the proofs for the main results.

Supplementary materials are available for download.