[Paper Review] Deep Neural Networks as Gaussian Processes

Jaehoon Lee, Yasaman Bahri+4 more|arXiv (Cornell University)|Nov 1, 2017

Gaussian Processes and Bayesian Inference16 references335 citations

TL;DR

The paper proves an exact equivalence between infinitely wide deep neural networks and Gaussian processes, and provides a scalable method to compute the corresponding GP kernel to enable Bayesian inference on regression tasks like MNIST and CIFAR-10.

ABSTRACT

It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. This correspondence enables exact Bayesian inference for infinite width neural networks on regression tasks by means of evaluating the corresponding GP. Recently, kernel functions which mimic multi-layer random neural networks have been developed, but only outside of a Bayesian framework. As such, previous work has not identified that these kernels can be used as covariance functions for GPs and allow fully Bayesian prediction with a deep neural network. In this work, we derive the exact equivalence between infinitely wide deep networks and GPs. We further develop a computationally efficient pipeline to compute the covariance function for these GPs. We then use the resulting GPs to perform Bayesian inference for wide deep neural networks on MNIST and CIFAR-10. We observe that trained neural network accuracy approaches that of the corresponding GP with increasing layer width, and that the GP uncertainty is strongly correlated with trained network prediction error. We further find that test performance increases as finite-width trained networks are made wider and more similar to a GP, and thus that GP predictions typically outperform those of finite-width networks. Finally we connect the performance of these GPs to the recent theory of signal propagation in random neural networks.

Motivation & Objective

• Establish an exact correspondence between deep, infinitely wide neural networks and Gaussian processes (GPs).
• Derive a recursive, deterministic kernel computation for deep network GPs across layers and nonlinearities.
• Show that Bayesian inference with the resulting GP can match or surpass finite-width neural networks on standard benchmarks.
• Demonstrate practical feasibility by applying the Neural Network GP (NNGP) to MNIST and CIFAR-10 and comparing to SGD-trained nets.
• Connect GP performance to signal propagation theory in random networks.

Proposed method

• Derive the NNGP kernel via a layerwise infinite-width limit using the central limit theorem.
• Define a recursive kernel update K^l(x, x') = σ_b^2 + σ_w^2 F_φ(K^{l-1}(x, x'), K^{l-1}(x, x), K^{l-1}(x', x')), where F_φ depends on the nonlinearity φ.
• Provide analytic forms for certain φ (e.g., arccosine kernel for ReLU) and numerical schemes for general φ to compute F_φ.
• Develop an efficient implementation to compute K^L with a preprocessing and bilinear-interpolation scheme to reduce complexity.
• Use Gaussian process regression with the derived kernel to perform exact Bayesian inference on regression targets, including uncertainty quantification.
• Relate kernel behavior to deep signal propagation theory and phase diagrams from random networks.

Experimental results

Research questions

• RQ1 Can deep, infinitely wide neural networks be exactly represented as Gaussian processes with a computable covariance kernel?
• RQ2 How does the depth L and choice of nonlinearity φ affect the GP kernel and predictive performance on image classification tasks?
• RQ3 Do GP posterior predictions with the NNGP kernel provide competitive or superior results compared to finite-width neural networks trained with SGD on datasets like MNIST and CIFAR-10?
• RQ4 How does the GP uncertainty correlate with actual prediction error on test data?
• RQ5 Is there a connection between NNGP performance and recent theories of signal propagation in random neural networks?

Key findings

• NNGPs often outperform finite-width networks trained with SGD on MNIST and CIFAR-10 under comparable settings.
• As network width increases, trained NNs begin to resemble the NNGP in performance, indicating a close link between SGD-trained nets and Bayesian inference in wide regimes.
• GP uncertainty estimates are highly correlated with actual prediction error on test data.
• Performance peaks align with regions predicted by deep signal propagation phase diagrams (ordered/chaotic phases) for different nonlinearities.
• GPs provide explicit, principled uncertainty measures for predictions, which is challenging for standard neural nets.