[Paper Review] Infinitely Wide Tensor Networks as Gaussian Process

Erdong Guo, David Draper|arXiv (Cornell University)|Jan 7, 2021

Gaussian Processes and Bayesian Inference32 references4 citations

TL;DR

This paper establishes that infinitely wide tensor networks—specifically pure matrix product states (MPS) and two hybrid architectures (neural kernel MPS and MPS with hidden neural layers)—converge to Gaussian processes (GPs) in the infinite-width limit. By deriving the mean and covariance functions of the induced GP, the authors demonstrate that hyperparameters such as prior standard deviations control the characteristic length scales of the GP, with numerical experiments confirming that increased prior variance leads to more complex sample paths.

ABSTRACT

Gaussian Process is a non-parametric prior which can be understood as a distribution on the function space intuitively. It is known that by introducing appropriate prior to the weights of the neural networks, Gaussian Process can be obtained by taking the infinite-width limit of the Bayesian neural networks from a Bayesian perspective. In this paper, we explore the infinitely wide Tensor Networks and show the equivalence of the infinitely wide Tensor Networks and the Gaussian Process. We study the pure Tensor Network and another two extended Tensor Network structures: Neural Kernel Tensor Network and Tensor Network hidden layer Neural Network and prove that each one will converge to the Gaussian Process as the width of each model goes to infinity. (We note here that Gaussian Process can also be obtained by taking the infinite limit of at least one of the bond dimensions $α_{i}$ in the product of tensor nodes, and the proofs can be done with the same ideas in the proofs of the infinite-width cases.) We calculate the mean function (mean vector) and the covariance function (covariance matrix) of the finite dimensional distribution of the induced Gaussian Process by the infinite-width tensor network with a general set-up. We study the properties of the covariance function and derive the approximation of the covariance function when the integral in the expectation operator is intractable. In the numerical experiments, we implement the Gaussian Process corresponding to the infinite limit tensor networks and plot the sample paths of these models. We study the hyperparameters and plot the sample path families in the induced Gaussian Process by varying the standard deviations of the prior distributions. As expected, the parameters in the prior distribution namely the hyper-parameters in the induced Gaussian Process controls the characteristic lengthscales of the Gaussian Process.

Motivation & Objective

• To investigate the functional limit of tensor networks as their width approaches infinity.
• To establish the equivalence between infinitely wide tensor networks and Gaussian processes (GPs).
• To analyze the mean and covariance functions of the resulting GP in a general setup.
• To explore how hyperparameters of the prior distribution affect the characteristic length scales of the induced GP.
• To validate theoretical findings through numerical experiments on sample path families under varying prior variances.

Proposed method

• Derive the infinite-width limit of pure matrix product states (MPS) and show convergence to a GP with a well-defined mean and covariance function.
• Extend the analysis to two hybrid architectures: neural kernel MPS and MPS with hidden neural layers, proving their convergence to GP in the infinite-width limit.
• Compute the mean vector and covariance matrix of the finite-dimensional distribution of the induced GP using expectation operators.
• Handle intractable integrals in the covariance function by deriving analytical approximations.
• Implement the induced GP from infinite-width tensor networks numerically and generate sample paths for visualization.
• Systematically vary the standard deviation of the prior distribution to study its effect on sample path complexity and length scales.

Experimental results

Research questions

• RQ1 Does the infinite-width limit of a pure matrix product state (MPS) tensor network converge to a Gaussian process?
• RQ2 How do hybrid tensor network architectures—specifically neural kernel MPS and MPS with hidden neural layers—behave in the infinite-width limit?
• RQ3 What are the explicit forms of the mean and covariance functions of the GP induced by infinitely wide tensor networks?
• RQ4 How do hyperparameters such as the standard deviation of the prior distribution affect the characteristic length scales of the resulting GP?
• RQ5 Can numerical experiments confirm that increasing prior variance leads to more complex and flexible sample paths in the induced GP?

Key findings

• The infinite-width limit of a pure matrix product state (MPS) converges to a Gaussian process, though the resulting GP is trivial with zero uncertainty bands due to the absence of non-linearity.
• The neural kernel MPS and MPS with hidden neural layers both converge to a Gaussian process as their width approaches infinity, extending the applicability of the GP equivalence beyond purely linear models.
• The mean function and covariance function of the induced GP are explicitly derived for a general setup, providing a closed-form characterization of the limiting process.
• The covariance function is approximated when the expectation integral is intractable, enabling practical computation of the GP kernel.
• Numerical experiments confirm that increasing the standard deviation of the prior distribution leads to more complex and higher-variability sample paths, directly controlling the characteristic length scales of the GP.
• Sample path families generated from the infinite-width tensor network models show that hyperparameters such as prior variance govern the complexity and smoothness of the function space, validating the theoretical control over GP properties.