[Paper Review] Gaussian Process Behaviour in Wide Deep Neural Networks

Alexander Matthews, Mark Rowland+3 more|arXiv (Cornell University)|Apr 30, 2018

Gaussian Processes and Bayesian InferenceComputer Science90 citations

TL;DR

This paper proves that deep, wide, fully connected neural networks with multiple hidden layers converge in distribution to a Gaussian process as widths grow, under mild conditions, and empirically validates this with MMD against Gaussian process analogues and exact Bayesian neural networks.

ABSTRACT

Whilst deep neural networks have shown great empirical success, there is still much work to be done to understand their theoretical properties. In this paper, we study the relationship between random, wide, fully connected, feedforward networks with more than one hidden layer and Gaussian processes with a recursive kernel definition. We show that, under broad conditions, as we make the architecture increasingly wide, the implied random function converges in distribution to a Gaussian process, formalising and extending existing results by Neal (1996) to deep networks. To evaluate convergence rates empirically, we use maximum mean discrepancy. We then compare finite Bayesian deep networks from the literature to Gaussian processes in terms of the key predictive quantities of interest, finding that in some cases the agreement can be very close. We discuss the desirability of Gaussian process behaviour and review non-Gaussian alternative models from the literature.

Motivation & Objective

• Extend the theoretical understanding of random fully connected networks with more than one hidden layer.
• Prove convergence of wide deep networks to Gaussian processes under broad conditions.
• Empirically evaluate convergence rates using maximum mean discrepancy (MMD).
• Compare finite Bayesian deep networks to Gaussian processes in predictive quantities.
• Discuss implications for Bayesian deep learning and initialization/dynamics.

Proposed method

• Define a fully connected network with D hidden layers and standard random normal priors on weights and biases.
• Scale weight variances with width following Neal (1996) to avoid variance explosion.
• Use the multivariate central limit theorem to show the joint distribution of layer activations converges to a multivariate normal, inducing a Gaussian process in the limit (Theorem 4).
• Impose a linear envelope property on the nonlinearity (|φ(u)| ≤ c + m|u|).
• Use a recursion lemma (Lemma 2) to characterize the limiting covariance structure across layers.
• Measure convergence to GP via maximum mean discrepancy (MMD) between finite networks and GP analogues.

Experimental results

Research questions

• RQ1 Under what conditions does a deep, wide neural network converge in distribution to a Gaussian process as width grows?
• RQ2 How does the choice of width growth (increasing per layer) affect convergence to the GP?
• RQ3 What is the rate of convergence to the Gaussian process in terms of depth and width?
• RQ4 How well do finite Bayesian deep networks agree with GP predictions for common datasets and priors?

Key findings

• A rigorous result (Theorem 4) shows convergence to a Gaussian process for any fixed number of hidden layers with strictly increasing width functions.
• The limiting GP has zero mean and a covariance determined by a recursion (Lemma 2).
• Empirical MMD experiments show finite networks increasingly resemble their GP analogues as width grows, with slower convergence for deeper nets.
• Among six datasets, five show close agreement between exact GP inference and finite Bayesian neural networks using MCMC.
• Different width growth schemes (identity, largest last, largest first) still lead to GP convergence as width increases, confirming independence from the specific width function shape under the theorem.