[Paper Review] Doubly Decomposing Nonparametric Tensor Regression

Masaaki Imaizumi, Kohei Hayashi|arXiv (Cornell University)|Jun 19, 2015

Tensor decomposition and applications28 references20 citations

TL;DR

This paper proposes Doubly Decomposing Nonparametric Tensor Regression (AMNR), a method that combines low-rank tensor decomposition with additive-multiplicative nonparametric modeling to reduce the curse of dimensionality in high-order tensor regression. By decomposing both the input tensor and the function space, AMNR achieves a faster convergence rate of $ O(n^{-eta/(2eta + \max_k I_k)}) $, significantly improving upon standard nonparametric rates while maintaining consistency with the same function class under smoothness conditions.

ABSTRACT

Nonparametric extension of tensor regression is proposed. Nonlinearity in a high-dimensional tensor space is broken into simple local functions by incorporating low-rank tensor decomposition. Compared to naive nonparametric approaches, our formulation considerably improves the convergence rate of estimation while maintaining consistency with the same function class under specific conditions. To estimate local functions, we develop a Bayesian estimator with the Gaussian process prior. Experimental results show its theoretical properties and high performance in terms of predicting a summary statistic of a real complex network.

Motivation & Objective

• Address the curse of dimensionality in nonparametric tensor regression, where convergence rates deteriorate due to high input dimensionality $ \prod_k I_k $.
• Overcome the limitations of linear tensor regression (TLR), which suffers from high bias when the true function is nonlinear or full-rank.
• Develop a nonparametric model that maintains expressive function class representation while drastically reducing effective dimensionality.
• Achieve faster convergence rates than standard nonparametric methods by exploiting structural decomposition in both input space and function space.
• Provide theoretical guarantees for posterior contraction and risk bounds under smoothness and regularity conditions.

Proposed method

• Propose Additive-Multiplicative Nonparametric Regression (AMNR), modeling the regression function as a sum of products of local functions applied to rank-one tensor components.
• Utilize CANDECOMP/PARAFAC (CP) decomposition to represent the input tensor $ X $ as a sum of rank-one tensors $ \sum_r \lambda_r \bigotimes_k x_r^{(k)} $, enabling component-wise modeling.
• Model each component function using a Gaussian Process (GP) prior, allowing flexible, nonparametric estimation with uncertainty quantification.
• Apply double decomposition: decompose the input tensor space via CP decomposition and the function space via additive-multiplicative expansion $ f = \sum_m \prod_k g_m^{(k)} $, reducing effective dimensionality.
• Establish posterior contraction rates using metric entropy and covering number arguments, showing improved convergence over standard nonparametric rates.
• Derive risk bounds by balancing approximation error (from truncating infinite series) and estimation error (from GP estimation), optimizing the truncation level $ M \asymp n^\zeta $.

Experimental results

Research questions

• RQ1 Can nonparametric tensor regression achieve faster convergence rates than standard nonparametric methods when the input is a high-dimensional tensor?
• RQ2 Does decomposing both the input tensor and the function space simultaneously improve estimation efficiency while preserving function class expressivity?
• RQ3 What is the optimal trade-off between approximation error and estimation error in a doubly decomposed nonparametric tensor model?
• RQ4 Can a Bayesian GP estimator with double decomposition achieve posterior contraction at a faster rate than standard nonparametric GP estimators?
• RQ5 How does the effective dimensionality of the problem scale in the proposed AMNR framework compared to the full tensor dimension?

Key findings

• The proposed AMNR model achieves a convergence rate of $ O(n^{-\beta/(2\beta + \max_k I_k)}) $, which is $ \prod_{k \neq k'} I_k $ times faster than the standard nonparametric rate $ O(n^{-\beta/(2\beta + \prod_k I_k)}) $, where $ k' = \arg\max_k I_k $.
• The model maintains the same function class as general nonparametric regression under smoothness conditions, ensuring consistency with the true function class.
• Posterior contraction of the GP estimator is established at rate $ \epsilon_n^{(k)} $, with covering number bounds showing that the effective complexity is reduced via double decomposition.
• The risk bound is minimized when the truncation level $ M \asymp n^{\zeta} $ with $ \zeta = \frac{\beta}{2\beta + \max_k I_k} \cdot \frac{1}{1 + \gamma} $, balancing approximation and estimation errors.
• Simulation results confirm the theoretical convergence rate, and real-world experiments on complex network summary statistics demonstrate high predictive performance.
• The method effectively mitigates the curse of dimensionality by reducing the effective input dimension to $ \max_k I_k $, rather than the full tensor dimension $ \prod_k I_k $.