[Paper Review] Computing low-rank approximations of large-scale matrices with the Tensor Network randomized SVD

Kim Batselier, Wenjian Yu+2 more|arXiv (Cornell University)|Jul 25, 2017

Tensor decomposition and applications19 references3 citations

TL;DR

This paper introduces the Tensor Network randomized SVD (TNrSVD) algorithm, a fast and accurate method for computing low-rank approximations of large-scale matrices in Matrix Product Operator (MPO) format. It achieves up to 17× speedup over state-of-the-art ALS-SVD and MALS-SVD methods while maintaining machine precision accuracy, and also proposes a novel matrix-to-MPO conversion algorithm up to 509× faster than TT-SVD.

ABSTRACT

We propose a new algorithm for the computation of a singular value decomposition (SVD) low-rank approximation of a matrix in the Matrix Product Operator (MPO) format, also called the Tensor Train Matrix format. Our tensor network randomized SVD (TNrSVD) algorithm is an MPO implementation of the randomized SVD algorithm that is able to compute dominant singular values and their corresponding singular vectors. In contrast to the state-of-the-art tensor-based alternating least squares SVD (ALS-SVD) and modified alternating least squares SVD (MALS-SVD) matrix approximation methods, TNrSVD can be up to 17 times faster while achieving the same accuracy. In addition, our TNrSVD algorithm also produces accurate approximations in particular cases where both ALS-SVD and MALS-SVD fail to converge. We also propose a new algorithm for the fast conversion of a sparse matrix into its corresponding MPO form, which is up to 509 times faster than the standard Tensor Train SVD (TT-SVD) method while achieving machine precision accuracy. The efficiency and accuracy of both algorithms are demonstrated in numerical experiments.

Motivation & Objective

• To address the computational bottleneck of low-rank approximation for large-scale matrices stored in MPO format.
• To overcome convergence failures of existing tensor-based SVD methods like ALS-SVD and MALS-SVD in certain cases.
• To develop a fast, accurate, and scalable matrix-to-MPO conversion algorithm for sparse matrices.
• To integrate randomized SVD techniques into the tensor network framework for improved performance.

Proposed method

• Proposes a randomized SVD algorithm tailored for matrices in Matrix Product Operator (MPO) format, leveraging tensor network structures.
• Adapts the randomized SVD framework to operate directly on MPO representations, avoiding full matrix formation.
• Uses power iterations and random projection to extract dominant singular values and vectors efficiently in MPO format.
• Introduces a novel matrix-to-MPO conversion algorithm using rank-revealing QR and tensor train rounding, achieving machine precision.
• Employs a Cuthill-McKee permutation to reduce MPO-rank before conversion, improving efficiency.
• Implements a rounding procedure with adaptive tolerances to control approximation error while preserving accuracy.

Experimental results

Research questions

• RQ1 Can a randomized SVD algorithm be effectively adapted to the tensor network framework for MPO-represented matrices to improve computational speed?
• RQ2 How does the performance of the proposed TNrSVD compare to ALS-SVD and MALS-SVD in terms of speed and accuracy for large-scale matrices?
• RQ3 Can the new matrix-to-MPO conversion algorithm achieve machine precision accuracy while being significantly faster than existing methods like TT-SVD?
• RQ4 In cases where ALS-SVD and MALS-SVD fail to converge, can TNrSVD still produce accurate low-rank approximations?
• RQ5 What is the impact of matrix permutation strategies on the resulting MPO-rank and computational efficiency?

Key findings

• The TNrSVD algorithm achieves up to 17× speedup over the MALS-SVD method while maintaining the same accuracy, as demonstrated on $2^N \times 2^N$ matrices.
• For rank-50 approximations of random matrices with prescribed singular values, TNrSVD was 17× faster than MALS-SVD and 5× faster than ALS-SVD.
• The proposed matrix-to-MPO conversion algorithm is up to 509× faster than the standard TT-SVD method, with machine precision accuracy.
• TNrSVD successfully computes accurate low-rank approximations in cases where ALS-SVD and MALS-SVD fail to converge, particularly for high-dimensional matrices.
• Standard matrix implementations of randomized SVD fail beyond $N=15$ due to memory constraints, while all tensor-based methods (including TNrSVD) remain scalable up to $N=50$.
• The use of adaptive rounding tolerances in TNrSVD ensures that the relative error in estimated singular values remains below $10^{-6}$ for rank-50 approximations.