[Paper Review] A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference

Kumar Shridhar, Felix Laumann+1 more|arXiv (Cornell University)|Jan 8, 2019

Generative Adversarial Networks and Image Synthesis54 references161 citations

TL;DR

This paper introduces Bayes by Backprop-based Bayesian CNNs with variational inference, using two convolutional operations for mean and variance to quantify uncertainty, and applies them to image classification, super-resolution, and GANs. It also discusses pruning and efficiency improvements.

ABSTRACT

Artificial Neural Networks are connectionist systems that perform a given task by learning on examples without having prior knowledge about the task. This is done by finding an optimal point estimate for the weights in every node. Generally, the network using point estimates as weights perform well with large datasets, but they fail to express uncertainty in regions with little or no data, leading to overconfident decisions. In this paper, Bayesian Convolutional Neural Network (BayesCNN) using Variational Inference is proposed, that introduces probability distribution over the weights. Furthermore, the proposed BayesCNN architecture is applied to tasks like Image Classification, Image Super-Resolution and Generative Adversarial Networks. The results are compared to point-estimates based architectures on MNIST, CIFAR-10 and CIFAR-100 datasets for Image CLassification task, on BSD300 dataset for Image Super Resolution task and on CIFAR10 dataset again for Generative Adversarial Network task. BayesCNN is based on Bayes by Backprop which derives a variational approximation to the true posterior. We, therefore, introduce the idea of applying two convolutional operations, one for the mean and one for the variance. Our proposed method not only achieves performances equivalent to frequentist inference in identical architectures but also incorporate a measurement for uncertainties and regularisation. It further eliminates the use of dropout in the model. Moreover, we predict how certain the model prediction is based on the epistemic and aleatoric uncertainties and empirically show how the uncertainty can decrease, allowing the decisions made by the network to become more deterministic as the training accuracy increases. Finally, we propose ways to prune the Bayesian architecture and to make it more computational and time effective.

Motivation & Objective

• Introduce Bayesian learning to CNNs to express predictive uncertainty and regularize training.
• Propose an efficient variational inference approach for CNN weights based on Bayes by Backprop.
• Show how to perform two convolutional operations (mean and variance) and apply local reparameterization to CNNs.
• Demonstrate uncertainty estimation (epistemic and aleatoric) and how uncertainty decreases with training.
• Explore pruning strategies (L1 regularization) to reduce parameters while maintaining accuracy.

Proposed method

• Adopt Bayes by Backprop to approximate the true posterior over CNN weights with a variational distribution q(w) .
• Represent weight uncertainty with Gaussian variational posteriors and learn mean and variance via two sequential convolutions (one for mean, one for variance).
• Apply the local reparameterization trick to CNNs to sample activations rather than weights, improving computational efficiency.
• Derive and optimize the variational free energy (KL divergence term plus expected log-likelihood) to train the model.
• Use L1 regularization to prune non-essential weights and fine-tune the pruned model to recover performance.
• Extend Bayesian CNNs to tasks beyond classification, including image super-resolution and Generative Adversarial Networks.

Experimental results

Research questions

• RQ1 Can Bayes by Backprop be efficiently applied to convolutional neural networks to produce calibrated uncertainty estimates?
• RQ2 How does the two-convolution approach (mean and variance) compare to point-estimate CNNs in terms of performance and regularization?
• RQ3 What is the impact of Bayesian CNNs on epistemic and aleatoric uncertainty in image-related tasks?
• RQ4 Can uncertainty-aware CNNs be pruned effectively without sacrificing accuracy, and how does this scale to SR and GAN tasks?
• RQ5 Do Bayesian CNNs offer competitive results on standard datasets (e.g., MNIST, CIFAR) relative to frequentist architectures?

Key findings

• Bayesian CNNs with variational inference can achieve performances equivalent to point-estimate architectures on comparable models.
• Uncertainty is decomposed into epistemic and aleatoric components, and uncertainty decreases as training accuracy improves, leading to more deterministic decisions.
• The two-convolutional-operations scheme enables learning both the mean and variance of weights without doubling the entire parameter count.
• The local reparameterization trick accelerates training by sampling activations instead of weights in convolutional layers.
• Pruning via L1 regularization reduces parameter count with minimal or no loss in predictive performance, improving model efficiency.
• The Bayesian framework is demonstrated across image classification, image super-resolution, and GAN tasks, with comparisons to non-Bayesian baselines.