[Paper Review] Quantum-Inspired Tensor Neural Networks for Partial Differential Equations

Raj Patel, Chia-Wei Hsing+12 more|arXiv (Cornell University)|Aug 3, 2022

Model Reduction and Neural Networks4 citations

TL;DR

This paper proposes Quantum-Inspired Tensor Neural Networks (TNNs) that leverage Tensor Network (TN) architectures—specifically Matrix Product Operators (MPOs)—to solve high-dimensional parabolic partial differential equations (PDEs) more efficiently than standard Dense Neural Networks (DNNs). By representing weight matrices via low-rank tensor decompositions, TNNs achieve comparable accuracy to DNNs with up to 66% fewer parameters and train up to 32.8% faster, as demonstrated on the Black-Scholes-Barenblatt and Hamilton-Jacobi-Bellman PDEs.

ABSTRACT

Partial Differential Equations (PDEs) are used to model a variety of dynamical systems in science and engineering. Recent advances in deep learning have enabled us to solve them in a higher dimension by addressing the curse of dimensionality in new ways. However, deep learning methods are constrained by training time and memory. To tackle these shortcomings, we implement Tensor Neural Networks (TNN), a quantum-inspired neural network architecture that leverages Tensor Network ideas to improve upon deep learning approaches. We demonstrate that TNN provide significant parameter savings while attaining the same accuracy as compared to the classical Dense Neural Network (DNN). In addition, we also show how TNN can be trained faster than DNN for the same accuracy. We benchmark TNN by applying them to solve parabolic PDEs, specifically the Black-Scholes-Barenblatt equation, widely used in financial pricing theory, empirically showing the advantages of TNN over DNN. Further examples, such as the Hamilton-Jacobi-Bellman equation, are also discussed.

Motivation & Objective

• To address the high memory and training cost of deep learning methods for solving high-dimensional PDEs.
• To overcome the limitations of standard Dense Neural Networks (DNNs), including parameter inefficiency and slow convergence.
• To demonstrate that Tensor Neural Networks (TNNs), inspired by quantum many-body physics, can outperform DNNs in both parameter efficiency and training speed.
• To rigorously quantify the advantage of TNNs by comparing them not to a single DNN, but to the entire family of DNNs with the same number of parameters.
• To validate the approach on benchmark PDEs, including the Black-Scholes-Barenblatt and Hamilton-Jacobi-Bellman equations.

Proposed method

• The method transforms standard DNNs into Tensor Neural Networks (TNNs) by replacing dense weight matrices with Matrix Product Operators (MPOs), a form of tensor network decomposition.
• MPOs represent weight matrices as a sequence of low-rank tensors, drastically reducing the number of trainable parameters while preserving representational capacity.
• The TNN architecture is trained using standard backpropagation, with the MPO structure enabling efficient gradient computation and memory savings.
• The authors compare TNNs not to a single DNN, but to all possible two-layer DNNs with the same total number of parameters, ensuring fair performance benchmarking.
• The approach is validated on the Black-Scholes-Barenblatt PDE and further tested on the Hamilton-Jacobi-Bellman equation in supplementary experiments.
• Convergence speed and parameter count are evaluated across multiple bond dimensions (χ) and network widths to analyze generalization and scalability.

Experimental results

Research questions

• RQ1 Can tensor network-based neural networks (TNNs) achieve the same accuracy as DNNs while using significantly fewer parameters?
• RQ2 Does the TNN architecture enable faster training convergence compared to DNNs with equivalent parameter counts?
• RQ3 To what extent can TNNs outperform the best-performing DNN architectures when both are constrained to the same number of parameters?
• RQ4 How does the bond dimension (χ) in the MPO representation affect the performance and convergence of TNNs relative to DNNs?
• RQ5 Can the TNN framework be effectively applied to real-world PDEs such as the Black-Scholes-Barenblatt and Hamilton-Jacobi-Bellman equations?

Key findings

• TNNs achieved the same solution accuracy as DNNs while using only 353 parameters, compared to 1057 for the best-matching DNN, representing a 66% parameter reduction.
• For the TNN(16) architecture with bond dimension χ=4, training converged 26.9% faster than the best-performing DNN with the same number of parameters.
• The largest training speedup of 32.8% was observed for TNN(144) with χ=4, outperforming all DNNs with the same parameter count.
• As the bond dimension increased, the performance gap between TNNs and DNNs diminished, confirming that MPOs can approximate dense matrices when χ is sufficiently large.
• The TNN framework consistently outperformed all DNNs with equivalent parameter counts across multiple configurations, validating its superiority in both efficiency and convergence speed.
• The method was successfully extended to the Hamilton-Jacobi-Bellman equation, demonstrating broader applicability beyond the Black-Scholes-Barenblatt case.