Abstract

Tensor regression has attracted significant attention in statistical research. This

study tackles the challenge of handling covariates with smooth varying structures. We introduce a novel framework, termed functional tensor regression, which incorporates both the

tensor and functional aspects of the covariate. To address the high dimensionality and functional continuity of the regression coefficient, we employ a low Tucker rank decomposition

along with smooth regularization for the functional mode.

We develop a functional Riemannian Gauss–Newton algorithm that demonstrates a provable quadratic convergence rate,

while the estimation error bound is based on the tensor covariate dimension. Simulations and

a neuroimaging analysis illustrate the finite sample performance of the proposed method.

Information

Preprint No.SS-2024-0342
Manuscript IDSS-2024-0342
Complete AuthorsTongyu Li, Fang Yao, Anru R. Zhang
Corresponding AuthorsFang Yao
Emailsfyao@math.pku.edu.cn

References

  1. Absil, P.-A., R. Mahony, and R. Sepulchre (2009). Optimization algorithms on matrix manifolds. Princeton University Press.
  2. Ahmed, T., H. Raja, and W. U. Bajwa (2020). Tensor regression using low-rank and sparse tucker decompositions. SIAM Journal on Mathematics of Data Science 2(4), 944–966.
  3. Bi, X., A. Qu, and X. Shen (2018). Multilayer tensor factorization with applications to recommender systems. The Annals of Statistics 46(6B), 3308–3333.
  4. Bi, X., X. Tang, Y. Yuan, Y. Zhang, and A. Qu (2021). Tensors in statistics. Annual review of statistics and its application 8, 345–368.
  5. Billio, M., R. Casarin, and M. Iacopini (2024). Bayesian markov-switching tensor regression for time-varying networks. Journal of the American Statistical Association 119(545), 109–121.
  6. Billio, M., R. Casarin, M. Iacopini, and S. Kaufmann (2023). Bayesian dynamic tensor regression. Journal of Business & Economic Statistics 41(2), 429–439.
  7. Boumal, N. (2023). An introduction to optimization on smooth manifolds. Cambridge University Press.
  8. Cai, T. T. and P. Hall (2006). Prediction in functional linear regression. The Annals of Statistics 34(5), 2159–2179.
  9. Chen, H., G. Raskutti, and M. Yuan (2019). Non-convex projected gradient descent for generalized low-rank tensor regression. The Journal of Machine Learning Research 20(1), 172–208.
  10. Chen, R., D. Yang, and C.-H. Zhang (2022). Factor models for high-dimensional tensor time series. Journal of the American Statistical Association 117(537), 94–116.
  11. Chen, Z., Y. Yang, and F. Yao (2024). Dynamic matrix recovery. Journal of the American Statistical Association 119(548), 2996–3007.
  12. Crambes, C., A. Kneip, and P. Sarda (2009). Smoothing splines estimators for functional linear regression. The Annals of Statistics 37(1), 35–72.
  13. De Lathauwer, L., B. De Moor, and J. Vandewalle (2000). A multilinear singular value decomposition. SIAM journal on Matrix Analysis and Applications 21(4), 1253–1278.
  14. Durham, T. J., M. W. Libbrecht, J. J. Howbert, J. Bilmes, and W. S. Noble (2018). Predictd parallel epigenomics data imputation with cloud-based tensor decomposition. Nature communications 9(1), 1–15.
  15. Eubank, R. L. (1999). Nonparametric regression and spline smoothing. CRC press.
  16. Gandy, S., B. Recht, and I. Yamada (2011). Tensor completion and low-n-rank tensor recovery via convex optimization. Inverse problems 27(2), 025010.
  17. Goldfarb, D. and Z. Qin (2014). Robust low-rank tensor recovery: Models and algorithms. SIAM Journal on Matrix Analysis and Applications 35(1), 225–253.
  18. Guhaniyogi, R. (2020). Bayesian methods for tensor regression. Wiley StatsRef: Statistics Reference Online, 1–18.
  19. Guhaniyogi, R., S. Qamar, and D. B. Dunson (2017). Bayesian tensor regression. Journal of Machine Learning Research 18(79), 1–31.
  20. Hackbusch, W. (2019). Tensor spaces and numerical tensor calculus (2 ed.), Volume 56. Springer.
  21. Hall, P. and J. L. Horowitz (2007). Methodology and convergence rates for functional linear regression. The Annals of Statistics 35(1), 70–91.
  22. Han, R., P. Shi, and A. R. Zhang (2024). Guaranteed functional tensor singular value decomposition. Journal of the American Statistical Association 119(546), 995–1007.
  23. Han, R., R. Willett, and A. R. Zhang (2022). An optimal statistical and computational framework for generalized tensor estimation. The Annals of Statistics 50(1), 1–29.
  24. Han, Y., D. Yang, C.-H. Zhang, and R. Chen (2024). Cp factor model for dynamic tensors. Journal of the Royal Statistical Society Series B: Statistical Methodology 86(5), 1383–1413.
  25. Hillar, C. J. and L.-H. Lim (2013). Most tensor problems are np-hard. Journal of the ACM (JACM) 60(6), 1–39.
  26. Hoff, P. D. (2015). Multilinear tensor regression for longitudinal relational data. The annals of applied statistics 9(3), 1169–1193.
  27. Jing, B.-Y., T. Li, Z. Lyu, and D. Xia (2021). Community detection on mixture multilayer networks via regularized tensor decomposition. The Annals of Statistics 49(6), 3181–3205.
  28. Kim, T.-K., S.-F. Wong, and R. Cipolla (2007). Tensor canonical correlation analysis for action classification. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE.
  29. Koch, O. and C. Lubich (2010). Dynamical tensor approximation. SIAM Journal on Matrix Analysis and Applications 31(5), 2360–2375.
  30. Kolda, T. G. and B. W. Bader (2009). Tensor decompositions and applications. SIAM review 51(3), 455–500.
  31. Li, L. and X. Zhang (2017). Parsimonious tensor response regression. Journal of the American Statistical Association 112(519), 1131–1146.
  32. Li, X., D. Xu, H. Zhou, and L. Li (2018). Tucker tensor regression and neuroimaging analysis. Statistics in Biosciences 10(3), 520–545.
  33. Liu, J., C. Zhu, Z. Long, Y. Liu, et al. (2021). Tensor regression. Foundations and Trends® in Machine Learning 14(4), 379–565.
  34. Lock, E. F. (2018). Tensor-on-tensor regression. Journal of Computational and Graphical Statistics 27(3), 638–647.
  35. Lu, C., J. Feng, Y. Chen, W. Liu, Z. Lin, and S. Yan (2019). Tensor robust principal component analysis with a new tensor nuclear norm. IEEE transactions on pattern analysis and machine intelligence 42(4), 925–938.
  36. Lu, H., K. N. Plataniotis, and A. N. Venetsanopoulos (2008). Mpca: Multilinear principal component analysis of tensor objects. IEEE transactions on Neural Networks 19(1), 18–39.
  37. Luo, Y., D. Tao, K. Ramamohanarao, C. Xu, and Y. Wen (2015). Tensor canonical correlation analysis for multi-view dimension reduction. IEEE transactions on Knowledge and Data Engineering 27(11), 3111–3124.
  38. Luo, Y. and A. R. Zhang (2023). Low-rank tensor estimation via riemannian gauss-newton: Statistical optimality and second-order convergence. Journal of Machine Learning Research 24(381), 1–48.
  39. Makantasis, K., A. D. Doulamis, N. D. Doulamis, and A. Nikitakis (2018). Tensor-based classification models for hyperspectral data analysis. IEEE Transactions on Geoscience and Remote Sensing 56(12), 6884–6898.
  40. Makris, N., S. L. Buka, J. Biederman, G. M. Papadimitriou, S. M. Hodge, E. M. Valera, A. B. Brown,
  41. G. Bush, M. C. Monuteaux, V. S. Caviness, et al. (2008). Attention and executive systems abnormalities in adults with childhood adhd: a dt-mri study of connections. Cerebral cortex 18(5), 1210–1220.
  42. Phan, A. H. and A. Cichocki (2010). Tensor decompositions for feature extraction and classification of high dimensional datasets. Nonlinear theory and its applications, IEICE 1(1), 37–68.
  43. Ramsay, J. and B. Silverman (2005). Functional data analysis (2 ed.). Springer.
  44. Ramsay, J. O. and C. Dalzell (1991). Some tools for functional data analysis. Journal of the Royal Statistical Society: Series B (Methodological) 53(3), 539–561.
  45. Rauhut, H., R. Schneider, and ˇZ. Stojanac (2017). Low rank tensor recovery via iterative hard thresholding. Linear Algebra and its Applications 523, 220–262.
  46. Sowell, E. R., P. M. Thompson, S. E. Welcome, A. L. Henkenius, A. W. Toga, and B. S. Peterson (2003). Cortical abnormalities in children and adolescents with attention-deficit hyperactivity disorder. The Lancet 362(9397), 1699–1707.
  47. Spencer, D., R. Guhaniyogi, R. Shinohara, and R. Prado (2022). Bayesian tensor regression using the tucker decomposition for sparse spatial modeling. arXiv preprint arXiv:2203.04733.
  48. Sun, W. W. and L. Li (2017). Store: sparse tensor response regression and neuroimaging analysis. The Journal of Machine Learning Research 18(1), 4908–4944.
  49. Sun, W. W. and L. Li (2019). Dynamic tensor clustering. Journal of the American Statistical Association 114(528), 1894–1907.
  50. Tucker, L. R. (1966). Some mathematical notes on three-mode factor analysis. Psychometrika 31(3), 279– 311.
  51. Uschmajew, A. and B. Vandereycken (2013). The geometry of algorithms using hierarchical tensors. Linear Algebra and its Applications 439(1), 133–166.
  52. Utreras, F. (1983). Natural spline functions, their associated eigenvalue problem. Numerische Mathematik 42(1), 107–117.
  53. Vannieuwenhoven, N., R. Vandebril, and K. Meerbergen (2012). A new truncation strategy for the higherorder singular value decomposition. SIAM Journal on Scientific Computing 34(2), A1027–A1052.
  54. Zhang, C., R. Han, A. R. Zhang, and P. M. Voyles (2020). Denoising atomic resolution 4d scanning transmission electron microscopy data with tensor singular value decomposition. Ultramicroscopy 219, 113123.
  55. Zhang, Y., X. Bi, N. Tang, and A. Qu (2021). Dynamic tensor recommender systems. The Journal of Machine Learning Research 22(1), 3032–3066.
  56. Zhou, H. and L. Li (2014). Regularized matrix regression. Journal of the Royal Statistical Society Series B: Statistical Methodology 76(2), 463–483.
  57. Zhou, H., L. Li, and H. Zhu (2013). Tensor regression with applications in neuroimaging data analysis. Journal of the American Statistical Association 108(502), 540–552.
  58. Zhou, J., W. W. Sun, J. Zhang, and L. Li (2023). Partially observed dynamic tensor response regression. Journal of the American Statistical Association 118(541), 424–439.
  59. Zhou, Y. and H.-G. M¨uller (2022). Network regression with graph laplacians. Journal of Machine Learning Research 23(320), 1–41.
  60. Zhou, Y., R. K. Wong, and K. He (2021). Tensor linear regression: Degeneracy and solution. IEEE Access 9, 7775–7788.
  61. Zhou, Y., R. K. Wong, and K. He (2024). Broadcasted nonparametric tensor regression. Journal of the Royal Statistical Society Series B: Statistical Methodology 86(5), 1197–1220.

Acknowledgments

This

research is supported by the National Natural Science Foundation of China (No.

12292981), the National Key Research and Developement Program of China (No.

2022YFA1003801), , the New Cornerstone Science foundation through the Xplorer

Prize, the LMAM and the Fundamental Research Funds for the Central Universities,

Peking University (LMEQF).

Supplementary Materials

All proofs of the technical results and additional numerical results are collected in an

online Supplementary Material. (.pdf file) The code and data are made available in

a GitHub repository (https://github.com/kellty/FTReg).

Supplementary materials are available for download.