[Paper Review] A Solution for Large Scale Nonlinear Regression with High Rank and Degree at Constant Memory Complexity via Latent Tensor Reconstruction

Sándor Szedmák, Anna Cichońska+3 more|arXiv (Cornell University)|May 4, 2020

Tensor decomposition and applications31 references4 citations

TL;DR

This paper proposes a latent tensor reconstruction (LTR) method for large-scale nonlinear regression with high-degree polynomials and high tensor rank, achieving constant memory complexity through incremental, gradient-based learning of rank-one tensor components. The approach enables efficient, scalable learning of complex nonlinear functions on large datasets with linear time complexity and superior accuracy compared to state-of-the-art methods like FM and KRR.

ABSTRACT

This paper proposes a novel method for learning highly nonlinear, multivariate functions from examples. Our method takes advantage of the property that continuous functions can be approximated by polynomials, which in turn are representable by tensors. Hence the function learning problem is transformed into a tensor reconstruction problem, an inverse problem of the tensor decomposition. Our method incrementally builds up the unknown tensor from rank-one terms, which lets us control the complexity of the learned model and reduce the chance of overfitting. For learning the models, we present an efficient gradient-based algorithm that can be implemented in linear time in the sample size, order, rank of the tensor and the dimension of the input. In addition to regression, we present extensions to classification, multi-view learning and vector-valued output as well as a multi-layered formulation. The method can work in an online fashion via processing mini-batches of the data with constant memory complexity. Consequently, it can fit into systems equipped only with limited resources such as embedded systems or mobile phones. Our experiments demonstrate a favorable accuracy and running time compared to competing methods.

Motivation & Objective

• Address the challenge of learning highly nonlinear, multivariate functions from large-scale datasets with high polynomial degrees and ranks.
• Overcome the memory and computational bottlenecks of traditional tensor-based methods by maintaining constant memory complexity during training.
• Enable efficient, online learning via mini-batch processing suitable for resource-constrained environments such as mobile devices or embedded systems.
• Extend the applicability of polynomial models beyond symmetric factorization (as in FM) to general, non-symmetric nonlinear functions.
• Demonstrate strong performance on both synthetic and real-world datasets, including multilabel image annotation tasks.

Proposed method

• Reformulate nonlinear function learning as a tensor reconstruction problem, leveraging the fact that continuous functions can be approximated by polynomials.
• Represent polynomials using a factorization into rank-one tensors via linear forms: $ f(\mathbf{x}) = \sum_{t=1}^{n_t} \prod_{d=1}^{n_d} \ell_d^{(t)}(\mathbf{x}) $, where each $ \ell_d^{(t)} $ is a linear form.
• Use a gradient-based optimization algorithm to learn the tensor components incrementally, ensuring linear time complexity in sample size, degree, rank, and input dimension.
• Implement the method in a mini-batch fashion to maintain constant memory complexity, enabling online and scalable learning.
• Extend the framework to classification, multi-view learning, and vector-valued outputs via appropriate loss functions and tensor structure.
• Integrate a multi-layered formulation to model hierarchical nonlinear interactions, analogous to deep neural networks but with algebraic tensor structure.

Experimental results

Research questions

• RQ1 Can a tensor-based method achieve constant memory complexity while learning high-degree, high-rank nonlinear functions from large-scale data?
• RQ2 How does the proposed latent tensor reconstruction method compare in accuracy and efficiency to existing approaches like Factorization Machines and Kernel Ridge Regression?
• RQ3 To what extent can the method scale to datasets with over 10 million examples while maintaining linear time complexity in polynomial degree?
• RQ4 Does the method outperform symmetric polynomial models (e.g., FM) on non-symmetric, complex nonlinear functions?
• RQ5 Can the method be effectively applied to real-world, high-sparsity multilabel prediction tasks such as image annotation?

Key findings

• The LTR method achieves significantly higher prediction accuracy than Factorization Machines (FM) and TFFM on synthetic data with high noise levels, polynomial degrees, and variable counts.
• On a small dataset (5,000 examples), LTR and TFFM underperform compared to Kernel Ridge Regression (KRR) due to high parameter space, but LTR maintains better scalability on larger datasets.
• LTR demonstrates linear time complexity in polynomial degree, as shown in experiments with up to 10 million samples, outperforming KRR, which exhibits significantly higher time complexity.
• On real-world multilabel image datasets (Corel5k, Espgame, Iaprtc12), LTR achieves F1 scores of 42.7%, 27.2%, and 37.0% respectively, outperforming prior methods like MBRM, TagProp, and FastTag.
• The method maintains constant memory complexity during training, enabling deployment on systems with limited resources such as mobile devices or embedded systems.
• The LTR algorithm achieves competitive training times, with execution time scaling linearly with polynomial degree, even at scale (10^7 examples), unlike KRR which scales poorly.