[Paper Review] Fundamental Tensor Operations for Large-Scale Data Analysis in Tensor Train Formats

Namgil Lee, Andrzej Cichocki|arXiv (Cornell University)|May 30, 2014

Tensor decomposition and applications38 references25 citations

TL;DR

This paper introduces extended multilinear tensor operations—such as contracted products, partial traces, and generalized Kronecker products—for tensor train (TT) formats, enabling efficient large-scale numerical analysis. By reformulating TT decompositions using these operations, the authors simplify index notation, prove orthonormality of frame matrices, and derive explicit representations of localized linear maps, significantly enhancing algorithmic design for high-dimensional systems of equations and eigenvalue problems.

ABSTRACT

We discuss extended definitions of linear and multilinear operations such as Kronecker, Hadamard, and contracted products, and establish links between them for tensor calculus. Then we introduce effective low-rank tensor approximation techniques including Candecomp/Parafac (CP), Tucker, and tensor train (TT) decompositions with a number of mathematical and graphical representations. We also provide a brief review of mathematical properties of the TT decomposition as a low-rank approximation technique. With the aim of breaking the curse-of-dimensionality in large-scale numerical analysis, we describe basic operations on large-scale vectors, matrices, and high-order tensors represented by TT decomposition. The proposed representations can be used for describing numerical methods based on TT decomposition for solving large-scale optimization problems such as systems of linear equations and symmetric eigenvalue problems.

Motivation & Objective

• Address the curse-of-dimensionality in high-order tensor computations by enabling efficient low-rank approximations using tensor train (TT) formats.
• Develop a coordinate-free, multilinear algebraic framework for TT decompositions to replace complex index notation.
• Enable practical numerical algorithms for large-scale systems of linear equations and symmetric eigenvalue problems via TT-based representations.
• Establish mathematical links between fundamental tensor operations (e.g., Kronecker, Hadamard, contracted products) and TT decompositions.
• Provide explicit, computationally tractable representations of localized linear maps and frame matrices essential for alternating linear schemes (ALS).

Proposed method

• Introduce generalized tensor operations such as the partial contracted product, partial trace, and partial Kronecker product for block tensors.
• Define tensor train decomposition using core tensors with matricized TT-cores, enabling low-parametric representation of high-order tensors.
• Represent large-scale vectors and matrices via vectorized TT decompositions with block matrices $\widetilde{\mathbf{X}}^{(n)}$.
• Derive matrix-by-vector and quadratic form operations using TT-based matrix multiplication via $\mathbf{X}^{\neq n}$, the product of all TT-cores except the $n$-th.
• Use the contracted product and mode-1 multiplication to express operations like $\mathbf{y} = \mathbf{X}^{\neq n} \mathbf{y}^{(n)}$, enabling efficient computation.
• Prove orthonormality of frame matrices using tensor operations, replacing prior index-based proofs with coordinate-free algebra.

Experimental results

Research questions

• RQ1 How can fundamental tensor operations be generalized to support efficient computation in tensor train formats?
• RQ2 Can multilinear operations such as contracted and Kronecker products be used to simplify and unify TT-based numerical algorithms?
• RQ3 How can the orthonormality of frame matrices in alternating linear schemes be proven using tensor operations rather than index notation?
• RQ4 What is the explicit representation of localized linear maps $\widetilde{\underline{\mathbf{A}}}_n$ and $\overline{\underline{\mathbf{A}}}_n$ in TT format?
• RQ5 How do the proposed tensor operations enable efficient solution of large-scale systems of linear equations and eigenvalue problems?

Key findings

• The proposed tensor operations, especially the contracted product and partial trace, allow for a coordinate-free representation of TT decompositions, simplifying complex index notation.
• The orthonormality of frame matrices used in alternating linear schemes (ALS) is rigorously proven using the proposed multilinear algebraic framework.
• Explicit representations of localized linear maps $\widetilde{\underline{\mathbf{A}}}_n$ and $\overline{\underline{\mathbf{A}}}_n$ are derived using TT-based matrix multiplication and contraction.
• Matrix-by-vector and quadratic form operations on large-scale vectors and matrices are efficiently computed via TT decomposition using $\mathbf{X}^{\neq n}$, reducing computational cost.
• The tensor network diagram for the quadratic form $\mathbf{x}^{(n)\top} \overline{\mathbf{A}}_n \mathbf{x}^{(n)}$ is visualized, showing connections between TT cores and core tensors.
• The framework enables algorithmic stability and rank adaptivity in TT-based solvers for high-dimensional problems, with potential for future convergence analysis.