[Paper Review] Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes

Roman Novak, Lechao Xiao+7 more|arXiv (Cornell University)|Oct 11, 2018

Gaussian Processes and Bayesian Inference169 citations

TL;DR

The paper proves that deep CNNs with many channels converge to Gaussian processes, extends NN-GP equivalence to CNNs with/without pooling, and introduces Monte Carlo methods to estimate corresponding kernels for intractable cases.

ABSTRACT

There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs). This equivalence enables, for instance, test set predictions that would have resulted from a fully Bayesian, infinitely wide trained FCN to be computed without ever instantiating the FCN, but by instead evaluating the corresponding GP. In this work, we derive an analogous equivalence for multi-layer convolutional neural networks (CNNs) both with and without pooling layers, and achieve state of the art results on CIFAR10 for GPs without trainable kernels. We also introduce a Monte Carlo method to estimate the GP corresponding to a given neural network architecture, even in cases where the analytic form has too many terms to be computationally feasible. Surprisingly, in the absence of pooling layers, the GPs corresponding to CNNs with and without weight sharing are identical. As a consequence, translation equivariance, beneficial in finite channel CNNs trained with stochastic gradient descent (SGD), is guaranteed to play no role in the Bayesian treatment of the infinite channel limit - a qualitative difference between the two regimes that is not present in the FCN case. We confirm experimentally, that while in some scenarios the performance of SGD-trained finite CNNs approaches that of the corresponding GPs as the channel count increases, with careful tuning SGD-trained CNNs can significantly outperform their corresponding GPs, suggesting advantages from SGD training compared to fully Bayesian parameter estimation.

Motivation & Objective

• Motivate understanding of priors over functions encoded by deep CNNs in the infinite-channel limit.
• Establish a theoretical NN-GP equivalence for CNNs with and without pooling under broad conditions.
• Quantify the role of pooling, weight sharing, and translation invariance in the infinite-width limit.
• Provide practical methods to compute or approximate CNN-GP kernels for architectures where analytic forms are infeasible.

Proposed method

• Derive that pre-activations in CNNs are Gaussian conditioned on previous activations, with a covariance given by an affine map A(K).
• Show that activation covariances K^l become deterministic as channel counts grow, via a C∘A map.
• Prove convergence in distribution to a GP with kernel K_infty^L obtained by iterating (C∘A) on K^0.
• Demonstrate that CNNs without pooling have identical NN-GP kernels to locally connected networks in the infinite-channel limit.
• Describe vectorization and projection readouts to transform CNN-GP outputs into GP kernels over classes, including exact forms for K_infty^L.
• Introduce a Monte Carlo method to estimate NN-GP kernels when analytic forms are intractable, via MC-GP kernel estimation.

Experimental results

Research questions

• RQ1 Does a deep CNN with many channels correspond to a Gaussian process in the infinite-channel limit?
• RQ2 Do CNNs with pooling differ in the infinite-channel limit from those without pooling?
• RQ3 How can one compute or approximate the GP kernel for CNN architectures where the analytic form is too complex?
• RQ4 What is the effect of weight sharing and translation equivariance on the Bayesian infinite-width CNN kernel?
• RQ5 Can readout strategies (vectorization or projection) yield GP kernels over classes that reflect common CNN classifiers?

Key findings

• CNNs with many channels converge to NN-GP behavior, yielding a Gaussian process prior over functions.
• Without pooling, the CNN-GP matches the locally connected network GP, implying pooling and equivariance have no infinite-channel effect in that regime.
• Translation equivariance does not alter the infinite-width Bayesian treatment, differentiating CNNs from FCNs in the infinite-channel limit.
• A Monte Carlo approach can estimate CNN-GP kernels when a closed-form analytic kernel is infeasible, enabling practical kernel computation for pooling architectures.
• Finite-width CNNs trained with SGD may outperform their corresponding CNN-GP in some settings, indicating benefits beyond the infinite-width Bayesian view.