Abstract

Transfer learning leverages knowledge from a source do

main to enhance estimation or prediction accuracy in a target

task. To strengthen data privacy protection when aggregating information across different sources and targets, differential privacy

offers a promising solution.

In this work, we propose a transfer learning framework for high-dimensional sparse partial linear

models with a novel differential privacy guarantee.

Our main

algorithm consists of two steps. The first step constructs a surrogate linear model by removing the non-linear component in the

target model.

The second step applies noisy gradient aggregation to transfer information from source domains while preserv-

ing privacy guarantees. Theoretically, we establish a nearly optimal error bound for the proposed transfer method in partial lin-

ear model estimation, while incurring an acceptable privacy cost.

Moreover, the debiased LASSO method is adopted to construct

confidence intervals.

Finally, we use an e-value based multiple

testing approach to control the false discovery rate. The effectiveness of our method is demonstrated through simulation studies

and further supported by its application to real-world data.

Key words and phrases: Transfer learning, partial linear model, RKHS, differential privacy, high-dimensional inference, e-Benjamini-Hochberg

Information

Preprint No.SS-2025-0202
Manuscript IDSS-2025-0202
Complete AuthorsZhengyu Zhu, Yibo Yan, Heng Lian, Riquan Zhang
Corresponding AuthorsRiquan Zhang
Emailszhangriquan@163.com

References

  1. Abadi, M., A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang
  2. (2016). Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 308–318.
  3. Auddy, A., T. T. Cai, and A. Chakraborty (2025). Minimax and adaptive transfer learning for nonparametric classification under distributed differential privacy constraints. Journal of the Royal Statistical Society Series B: Statistical Methodology, qkaf070.
  4. Bastani, H. (2021). Predicting with proxies: Transfer learning in high dimension. Management Science 67(5), 2964–2984.
  5. Benjamini, Y. and Y. Hochberg (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B: Statistical Methodology 57(1), 289–300.
  6. Bickel, P. J., C. A. Klaassen, P. J. Bickel, Y. Ritov, J. Klaassen, J. A. Wellner, and
  7. Y. Ritov (1993). Efficient and Adaptive Estimation for Semiparametric Models, Volume 4. Springer.
  8. Cai, T., M. Li, and M. Liu (2025). Semi-supervised triply robust inductive transfer learning. Journal of the American Statistical Association 120(550), 1037–1047.
  9. Cai, T. T., A. Chakraborty, and L. Vuursteen (2024). Federated nonparametric hypothesis testing with differential privacy constraints: Optimal rates and adaptive tests. arXiv preprint arXiv:2406.06749.
  10. Cai, T. T., A. Chakraborty, and L. Vuursteen (2026). Optimal federated learning for nonparametric regression with heterogeneous distributed differential privacy constraints. Journal of the American Statistical Association, in press.
  11. Cai, T. T., Y. Wang, and L. Zhang (2021). The cost of privacy: Optimal rates of convergence for parameter estimation with differential privacy. The Annals of Statistics 49(5), 2825–2850.
  12. Cai, Z., S. Li, X. Xia, and L. Zhang (2026). Differentially private estimation and inference in high-dimensional regression with FDR control. Journal of Machine Learning Research 27, 1–54.
  13. Cui, S., X. Guo, and Z. Zhang (2025). Estimation and inference in ultrahighdimensional partially linear single-index models. Science China Mathematics 68(8), 1807–1840.
  14. Duchi, J. C., M. I. Jordan, and M. J. Wainwright (2018). Minimax optimal procedures for locally private estimation. Journal of the American Statistical Association 113(521), 182–201.
  15. Dwork, C., A. Roth, et al. (2014). The algorithmic foundations of differential privacy. Foundations and Trends ® in Theoretical Computer Science 9(3–4), 211–407.
  16. Dwork, C., W. Su, and L. Zhang (2021). Differentially private false discovery rate control. Journal of Privacy and Confidentiality 11(2).
  17. Engle, R. F., C. W. Granger, J. Rice, and A. Weiss (1986). Semiparametric estimates of the relation between weather and electricity sales. Journal of the American statistical Association 81(394), 310–320.
  18. Hanneke, S. and S. Kpotufe (2019). On the value of target data in transfer learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Advances in Neural Information Processing Systems, Volume 32. Curran
  19. Associates, Inc.
  20. He, B., H. Liu, X. Zhang, and J. Huang (2024). Representation transfer learning for semiparametric regression. arXiv preprint arXiv:2406.13197.
  21. Hu, X. and X. Zhang (2023). Optimal parameter-transfer learning by semiparametric model averaging. Journal of Machine Learning Research 24(358), 1–53.
  22. Jiao, Y., H. Lin, Y. Luo, and J. Z. Yang (2024). Deep transfer learning: Model framework and error analysis. arXiv preprint arXiv:2410.09383.
  23. Kim, S., D. Zeng, and J. M. Taylor (2017). Joint partially linear model for longitudinal data with informative drop-outs. Biometrics 73(1), 72–82.
  24. Kosorok, M. R. (2008). Introduction to Empirical Processes and Semiparametric Inference, Volume 61. Springer.
  25. Li, M., Y. Tian, Y. Feng, and Y. Yu (2024). Federated transfer learning with differential privacy. arXiv preprint arXiv:2403.11343.
  26. Li, N., Y. Fei, and X. Zhang (2024). Partial linear model averaging prediction for longitudinal data. Journal of Systems Science and Complexity 37(2), 863–885.
  27. Li, S., T. T. Cai, and H. Li (2022). Transfer learning for high-dimensional linear regression: Prediction, estimation and minimax optimality. Journal of the Royal Statistical Society Series B: Statistical Methodology 84(1), 149–173.
  28. Li, S., T. T. Cai, and H. Li (2023). Transfer learning in large-scale Gaussian graphical models with false discovery rate control. Journal of the American Statistical Association 118(543), 2171–2183.
  29. Lian, H. (2020). Asymptotics of the non-parametric function for b-splines-based estimation in partially linear models. International Statistical Review 88(1), 142–154.
  30. Lian, H., K. Zhao, and S. Lv (2019). Projected spline estimation of the nonparametric function in high-dimensional partially linear models for massive data. The Annals of Statistics 47(5), 2922–2949.
  31. Ling, N., Y. Yang, and Q. Peng (2025). Partial linear quantile regression model with incompletely observed functional covariates. Journal of Nonparametric Statistics 37(3), 713–739.
  32. Liu, W., X. Mao, and X. Zhang (2022). Fast and robust sparsity learning over networks: A decentralized surrogate median regression approach. IEEE Transactions on Signal Processing 70, 797–809.
  33. Liu, Y., S. Zhang, S. Ma, and Q. Zhang (2020). Tests for regression coefficients in high dimensional partially linear models. Statistics & Probability Letters 163, 108772.
  34. Ning, Y. and H. Liu (2017). A general theory of hypothesis tests and confidence regions for sparse high dimensional models. The Annals of Statistics 45(1), 158–195.
  35. Pournaderi, M. and Y. Xiang (2021). Differentially private variable selection via the knockoff filter. In 2021 IEEE 31st International Workshop on Machine Learning for Signal Processing, pp. 1–6. IEEE.
  36. Qiao, D. and Y.-X. Wang (2023). Offline reinforcement learning with differential privacy. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Advances in Neural Information Processing Systems, Volume 36, pp. 61395– 61436. Curran Associates, Inc.
  37. Ren, Z. and R. F. Barber (2024). Derandomised knockoffs: Leveraging e-values for false discovery rate control. Journal of the Royal Statistical Society Series B: Statistical Methodology 86(1), 122–154.
  38. Shi, H., W. Yang, N. Zhou, and X. Guo (2026). Inference for partially linear quantile regression models in ultrahigh dimension. Communications in Mathematics and Statistics 14(3), 495–540.
  39. Shi, Y., M. Hao, Y. Tang, and X. Guo (2025). Estimation and inference of highdimensional partially linear regression models with latent factors. arXiv preprint arXiv:2501.06529.
  40. Tan, F., X. Jiang, X. Guo, and L. Zhu (2021). Testing heteroscedasticity for regression models based on projections. Statistica Sinica 31(2), 625–646.
  41. Tian, Y. and Y. Feng (2023). Transfer learning under high-dimensional generalized linear models. Journal of the American Statistical Association 118(544), 2684–2697.
  42. Torrey, L. and J. Shavlik (2010). Transfer learning. In Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques, pp. 242–264. IGI global.
  43. Wainwright, M. J. (2019). High-Dimensional Statistics: A Non-Asymptotic Viewpoint, Volume 48. Cambridge University Press.
  44. Wang, D., L. Hu, H. Zhang, M. Gaboardi, and J. Xu (2023). Generalized linear models in non-interactive local differential privacy with public data. Journal of Machine Learning Research 24(132), 1–57.
  45. Wang, F. and Y. Yu (2025). Transfer learning for piecewise-constant mean estimation: Optimality, l1-and l0-penalization. Biometrika, asaf018.
  46. Wang, R. and A. Ramdas (2022). False discovery rate control with e-values. Journal of the Royal Statistical Society Series B: Statistical Methodology 84(3), 822–852.
  47. Wang, Y. and A. Nedić (2023). Tailoring gradient methods for differentially private distributed optimization. IEEE Transactions on Automatic Control 69(2), 872–887.
  48. Wei, K., J. Li, M. Ding, C. Ma, H. H. Yang, F. Farokhi, S. Jin, T. Q. Quek, and
  49. H. V. Poor (2020). Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security 15, 3454–3469.
  50. Wong, R. K., Y. Li, and Z. Zhu (2019). Partially linear functional additive models for multivariate functional data. Journal of the American Statistical Association 114(525), 406–418.
  51. Xia, X. and Z. Cai (2023). Adaptive false discovery rate control with privacy guarantee. Journal of Machine Learning Research 24(252), 1–35.
  52. Xie, H. and J. Huang (2009). SCAD-penalized regression in high-dimensional partially linear models. The Annals of Statistics 37(2), 673–696.
  53. Yao, Y. and G. Doretto (2010). Boosting for transfer learning with multiple sources. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1855–1862. IEEE.
  54. Yu, C., R. Ming, M. Xiao, and Z. Wang (2024). A flexible approach: Variable selection procedures with multilayer FDR control via e-values. arXiv preprint arXiv:2409.17039.
  55. Zhang, C.-H. and S. S. Zhang (2014). Confidence intervals for low dimensional parameters in high dimensional linear models. Journal of the Royal Statistical Society Series B: Statistical Methodology 76(1), 217–242.
  56. Zhang, Y. and Z. Zhu (2025). Transfer learning for high-dimensional quantile regression via convolution smoothing. Statistica Sinica 35, 939–958.
  57. Zhang, Z., R. Nakada, and L. Zhang (2024). Differentially private federated learning: Servers trustworthiness, estimation, and statistical inference. arXiv preprint arXiv:2404.16287.
  58. Zhao, F., N. Lin, and B. Zhang (2023). A new test for high-dimensional regression coefficients in partially linear models. Canadian Journal of Statistics 51(1), 5–18.
  59. Zhu, L., M. Ding, V. Aggarwal, J. Xu, and D. Wang (2024). Improved analysis of sparse linear regression in local differential privacy model. In 12th International Conference on Learning Representations.
  60. Zhu, Y. (2017). Nonasymptotic analysis of semiparametric regression models with high-dimensional parametric coefficients. The Annals of Statistics 45(5), 2274–2298.
  61. Zhu, Y., Z. Yu, and G. Cheng (2019). High dimensional inference in partially linear models. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2760–2769. PMLR.
  62. Zhu, Z., Y. Yan, L. Gefei, and R. Zhang (2025). Recent developments on statistical transfer learning. International Statistical Review, in press.

Acknowledgments

We would like to thank the Editor, the Associate Editor, and the two anonymous

reviewers for their valuable comments and constructive suggestions, which led to significant improvements in the paper. Yibo Yan’s research is supported by the National

Natural Science Foundation of China (12401390). Riquan Zhang’s research is supported

by the National Natural Science Foundation of China (12371272, 12531013).

Supplementary Materials

The online Supplementary Material contains some theoretical statements, auxiliary

results, all technical proofs and additional simulation results. The code supporting this

paper is available at https://github.com/moondanced/Trans-DPPLM.

Supplementary materials are available for download.