[Paper Review] Supervised Learning with Quantum-Inspired Tensor Networks

E. Miles Stoudenmire, David J. Schwab|arXiv (Cornell University)|May 18, 2016

Computational Physics and Python Applications40 references102 citations

TL;DR

This paper uses matrix product state (tensor train) representations to parameterize classifiers for supervised learning, achieving MNIST test error below 1% and offering a physics-inspired regularization perspective.

ABSTRACT

Tensor networks are efficient representations of high-dimensional tensors which have been very successful for physics and mathematics applications. We demonstrate how algorithms for optimizing such networks can be adapted to supervised learning tasks by using matrix product states (tensor trains) to parameterize models for classifying images. For the MNIST data set we obtain less than 1% test set classification error. We discuss how the tensor network form imparts additional structure to the learned model and suggest a possible generative interpretation.

Motivation & Objective

• Introduce a tensor-network approach to supervised learning using matrix product state (MPS) decompositions to parameterize classifiers.
• Map input data via a local feature map and represent the weight tensor as an MPS to enable scalable optimization.
• Demonstrate practical performance on MNIST and analyze how the tensor-network form regularizes learning and reveals structure.
• Provide interpretation of the learned model in terms of feature selection and potential generative viewpoints.

Proposed method

• Map inputs with a local feature map to create a high-dimensional tensor Phi(x) as a product state (rank-1) over N components.
• Represent the weight tensor W for all labels as a matrix product state (MPS) with bond dimension m to control capacity and regularization.
• Optimize the MPS weights using a sweeping algorithm inspired by DMRG, updating two neighboring tensors at a time and using gradient-like updates and SVD-based re-orthogonalization.
• Adaptively truncate the bond dimension via SVD to keep the MPS compact while maintaining optimality of the approximation.
• Scale the training and evaluation cost to be linear in the input space size, with cost scaling as d^3 m^3 N N_L N_T.
• Discuss a one-vs-all setup where f^ℓ(x) = W^ℓ · Phi(x) and classify by the label with the largest |f^ℓ(x)|.

Experimental results

Research questions

• RQ1 Can a tensor-network parameterization (specifically an MPS) efficiently represent and optimize the weight tensor for supervised classification tasks?
• RQ2 How does the MPS structure affect training efficiency, generalization, and the capacity to capture correlations in image data?
• RQ3 What performance gains are achievable on MNIST using an MPS-based classifier with varying bond dimension m?
• RQ4 What are the interpretive implications of the MPS form for feature selection and potential generative perspectives?

Key findings

• On MNIST (28x28 resized to 14x14), the MPS classifier achieves 0.97% test error with bond dimension m = 120 after three sweeps.
• Lower bond dimensions yield higher errors, e.g., m = 10 around 5% and m = 20 around 2% Test error.
• The training and evaluation cost scales linearly with input size and the approach allows adaptive compression by truncating around large singular values.
• The method integrates local feature maps and MPS structure to regularize learning beyond the feature map choice, offering a potential generative interpretation.
• The framework supports parallel optimization of internal tensors and provides a structured view of the learned features via the U/V orthogonal decompositions in the MPS.