[Paper Review] Bayesian Recurrent Neural Networks

Meire Fortunato, Charles Blundell+1 more|arXiv (Cornell University)|Apr 10, 2017

Adversarial Robustness in Machine LearningComputer Science52 references149 citations

TL;DR

The paper applies Bayes by Backprop to RNNs, introduces posterior sharpening, and shows improved language modeling and image captioning with uncertainty estimates.

ABSTRACT

In this work we explore a straightforward variational Bayes scheme for Recurrent Neural Networks. Firstly, we show that a simple adaptation of truncated backpropagation through time can yield good quality uncertainty estimates and superior regularisation at only a small extra computational cost during training, also reducing the amount of parameters by 80\%. Secondly, we demonstrate how a novel kind of posterior approximation yields further improvements to the performance of Bayesian RNNs. We incorporate local gradient information into the approximate posterior to sharpen it around the current batch statistics. We show how this technique is not exclusive to recurrent neural networks and can be applied more widely to train Bayesian neural networks. We also empirically demonstrate how Bayesian RNNs are superior to traditional RNNs on a language modelling benchmark and an image captioning task, as well as showing how each of these methods improve our model over a variety of other schemes for training them. We also introduce a new benchmark for studying uncertainty for language models so future methods can be easily compared.

Motivation & Objective

• Introduce a straightforward variational Bayes scheme for training RNNs (BBB) to capture weight uncertainty.
• Adapt truncated backpropagation through time to obtain posterior estimates with KL regularisation.
• Propose posterior sharpening to locally adapt the posterior per batch using gradient information.
• Demonstrate improved performance over traditional regularisation on language modeling and image captioning tasks.
• Provide a new benchmark for uncertainty in language models.

Proposed method

• Apply Bayes by Backprop to RNNs by sampling weights from a Gaussian posterior with diagonal covariance.
• Derive variational free energy for RNNs with truncated BPTT and distribute KL cost across minibatches and sequence cuts.
• Introduce posterior sharpening: a hierarchical posterior q(theta|varphi,(x,y)) that uses batch gradient information to adjust theta.
• Parameterise the sharpened posterior q(theta|varphi,(x,y)) via theta = varphi - eta * grad_theta log p(y|varphi,x) with eta learned.
• Train with Monte Carlo estimates and include KL regularisation terms in the training objective.
• Demonstrate improvements on language modeling (Penn Treebank) and image captioning (MSCOCO) benchmarks over baseline regularisation techniques.

Experimental results

Research questions

• RQ1 Does applying Bayes by Backprop to RNNs improve predictive performance and model calibration compared to standard regularisation?
• RQ2 Can posterior sharpening reduce gradient variance and improve learning in Bayesian RNNs?
• RQ3 How do Bayesian RNNs perform on language modeling and image captioning tasks relative to existing Bayesian and non-Bayesian approaches?
• RQ4 What are the uncertainty properties of Bayesian RNNs on out-of-distribution data?
• RQ5 Can the proposed methods be generalized to other neural architectures beyond RNNs?

Key findings

• Bayesian RNNs with BBB achieve competitive perplexities on Penn Treebank compared to dropout baselines.
• Posterior sharpening further lowers perplexity and improves calibration compared to standard BBB.
• BBB improves image captioning metrics (BLUE-4, CIDEr) over Show and Tell baseline in MSCOCO.
• Weight pruning shows many weights can be removed with limited impact on performance (about 80%).
• BBB provides uncertainty estimates that reflect calibration, outperforming MC-Dropout in entropy-based analyses.