[Paper Review] Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference

Yarin Gal, Zoubin Ghahramani|arXiv (Cornell University)|Jun 6, 2015

Gaussian Processes and Bayesian Inference28 references614 citations

TL;DR

This paper proposes a computationally efficient Bayesian convolutional neural network (CNN) that uses Bernoulli variational inference to place probability distributions over CNN kernels, enabling uncertainty estimation and improved robustness to overfitting on small datasets. By casting dropout training as approximate Bayesian inference, the method enables Monte Carlo dropout inference with no additional parameters, achieving state-of-the-art performance on CIFAR-10 with improved generalization and reduced overfitting.

ABSTRACT

Convolutional neural networks (CNNs) work well on large datasets. But labelled data is hard to collect, and in some applications larger amounts of data are not available. The problem then is how to use CNNs with small data -- as CNNs overfit quickly. We present an efficient Bayesian CNN, offering better robustness to over-fitting on small data than traditional approaches. This is by placing a probability distribution over the CNN's kernels. We approximate our model's intractable posterior with Bernoulli variational distributions, requiring no additional model parameters. On the theoretical side, we cast dropout network training as approximate inference in Bayesian neural networks. This allows us to implement our model using existing tools in deep learning with no increase in time complexity, while highlighting a negative result in the field. We show a considerable improvement in classification accuracy compared to standard techniques and improve on published state-of-the-art results for CIFAR-10.

Motivation & Objective

• To address the problem of overfitting in CNNs when training on small datasets with limited labeled data.
• To develop a Bayesian CNN that offers uncertainty estimates and better generalization without increasing model complexity.
• To provide a theoretically grounded justification for using dropout in convolutional layers, which is often avoided in practice due to performance degradation.
• To demonstrate that Monte Carlo dropout can be used as a valid approximation to Bayesian inference in CNNs, improving test accuracy.

Proposed method

• Uses Bernoulli distributions as variational approximations to the posterior over CNN kernel weights, requiring no additional parameters.
• Reinterprets standard dropout training as approximate Bayesian inference in a Bayesian neural network, specifically using variational inference with Bernoulli distributions.
• Applies Monte Carlo dropout at test time by averaging multiple forward passes through the network with dropout active.
• Implements the Bayesian CNN using standard deep learning tools by placing dropout after every convolutional layer during training.
• Derives the connection between dropout and variational inference, showing that dropout approximates marginalization over kernel weights.
• Employs existing deep learning frameworks to train the model with no change in training time or computational overhead.

Experimental results

Research questions

• RQ1 Why does standard dropout fail to improve performance when applied after convolutional layers in CNNs?
• RQ2 Can dropout be formally interpreted as approximate Bayesian inference in Bayesian neural networks?
• RQ3 Does Monte Carlo dropout provide a statistically valid approximation to Bayesian inference in CNNs, especially when applied after convolutional layers?
• RQ4 Can a Bayesian CNN with Bernoulli variational inference achieve better generalization and uncertainty estimation on small datasets compared to standard CNNs?
• RQ5 What is the optimal number of Monte Carlo samples required for stable performance improvement in test-time inference?

Key findings

• The proposed Bayesian CNN with MC dropout achieves a test error of 7.51 on the augmented CIFAR-10 dataset, representing a significant improvement over standard techniques.
• MC dropout reduces test error by more than one standard deviation after only 20 forward passes, with convergence observed by 100 samples.
• The method improves performance across multiple architectures (NIN, DSN, Augmented-DSN), with the largest gain observed in the Augmented-DSN model.
• The approach provides a mathematically grounded solution to the failure of standard dropout in convolutional layers, resolving a known negative result in the field.
• No additional parameters are required for the variational posterior, and training time remains identical to standard models, with only a marginal increase in test-time cost due to averaging.
• The method does not improve performance on ImageNet, suggesting that large datasets may already provide sufficient regularization, but could be beneficial for smaller subsets of such data.