[Paper Review] Non-asymptotic approximations of neural networks by Gaussian processes

Ronen Eldan, Dan Mikulincer+1 more|arXiv (Cornell University)|Feb 17, 2021

Markov Chains and Monte Carlo Methods30 references5 citations

TL;DR

This paper provides the first non-asymptotic, quantitative bounds on how well wide but finite-width neural networks can be approximated by Gaussian processes when initialized with random weights. By leveraging functional central limit theorems in infinite-dimensional spaces using transportation distances, it establishes convergence rates that depend on activation function properties—polynomial degree for polynomial activations and smoothness for general activations—offering a rigorous finite-width analog to Neal's classical asymptotic result.

ABSTRACT

We study the extent to which wide neural networks may be approximated by Gaussian processes when initialized with random weights. It is a well-established fact that as the width of a network goes to infinity, its law converges to that of a Gaussian process. We make this quantitative by establishing explicit convergence rates for the central limit theorem in an infinite-dimensional functional space, metrized with a natural transportation distance. We identify two regimes of interest; when the activation function is polynomial, its degree determines the rate of convergence, while for non-polynomial activations, the rate is governed by the smoothness of the function.

Motivation & Objective

• To close the gap between asymptotic results (infinite-width limits) and practical, finite-width neural networks by providing quantitative approximation bounds.
• To establish a functional central limit theorem (CLT) in an infinite-dimensional function space, rather than just finite-dimensional marginals.
• To derive explicit convergence rates for the approximation of randomly initialized wide neural networks by Gaussian processes, using transportation metrics.
• To show that the rate of convergence depends on the activation function: polynomial degree for polynomial activations and smoothness for general activations.
• To unify and extend prior work by reducing complex functional approximation problems to finite-dimensional settings via structural exploitation of neural network architecture.

Proposed method

• The authors model a two-layer neural network with random initialization as a random process on the unit sphere, using i.i.d. standard Gaussian weights and Rademacher signs.
• They define a transportation distance (Wasserstein-2) on the space of random processes in $ L^2(\mathbb{S}^{n-1}) $, enabling comparison of the network's law to a limiting Gaussian process.
• For polynomial activations, they embed the network into a high-dimensional tensor space and apply recent advances in high-dimensional CLT with explicit error bounds.
• For general smooth activations, they use polynomial approximation of the activation function and bound the error in terms of the smoothness of the activation, reducing the problem to the polynomial case.
• They apply Hille's theorem and Hermite polynomial expansions to control the decay of Fourier coefficients of the activation function.
• The final bounds are derived by combining error terms from approximation theory and probabilistic concentration, yielding explicit rates in terms of network width $k$, input dimension $n$, and activation smoothness.

Experimental results

Research questions

• RQ1 How fast does a randomly initialized wide neural network converge to a Gaussian process in function space, rather than just at finite sets of inputs?
• RQ2 What determines the convergence rate in the non-asymptotic regime—activation function properties or network width?
• RQ3 Can a functional CLT be established with explicit, quantitative bounds in an infinite-dimensional space using transportation metrics?
• RQ4 How does the rate of convergence differ between polynomial and non-polynomial activation functions?
• RQ5 To what extent can the analysis of general activations be reduced to the polynomial case via approximation theory?

Key findings

• For two-layer networks with polynomial activation of degree $d$, the Wasserstein-2 distance between the network and its limiting Gaussian process is bounded by $O(k^{-1/6})$ plus a term depending on the $L^2$-norm of the activation's Fourier coefficients.
• For ReLU activation, the convergence rate is $O\left(\left(\frac{1}{k^{1/6}} + \left(\frac{\log n \log \log k}{\log k}\right)^2\right)\right)$, with the $k^{-1/6}$ term dominating.
• For smooth activations like tanh, the convergence rate is exponentially fast: $O\left(\exp\left(-C^{-1}\sqrt{\frac{\log k}{\log n \log \log k}}\right)\right)$, significantly faster than polynomial rates.
• The convergence rate does not deteriorate with the number of input points, making the bounds robust for joint distributions over arbitrary finite sets of inputs.
• The analysis shows that the smoothness of the activation function governs the rate for non-polynomial cases, with $R_\sigma(d) \leq e^{-C\sqrt{d}}$ for $\tanh$, leading to super-polynomial convergence.
• The results establish a finite-width, quantitative analog to Neal’s classical CLT, providing the first functional, non-asymptotic approximation of wide neural networks by Gaussian processes.