[Paper Review] Dropout Inference in Bayesian Neural Networks with Alpha-divergences

Yingzhen Li, Yarin Gal|arXiv (Cornell University)|Mar 8, 2017

Adversarial Robustness in Machine Learning39 references107 citations

TL;DR

This paper reparameterizes alpha-divergence objectives to enable practical, dropout-based Bayesian neural network inference that improves uncertainty estimates and robustness to adversarial examples.

ABSTRACT

To obtain uncertainty estimates with real-world Bayesian deep learning models, practical inference approximations are needed. Dropout variational inference (VI) for example has been used for machine vision and medical applications, but VI can severely underestimates model uncertainty. Alpha-divergences are alternative divergences to VI's KL objective, which are able to avoid VI's uncertainty underestimation. But these are hard to use in practice: existing techniques can only use Gaussian approximating distributions, and require existing models to be changed radically, thus are of limited use for practitioners. We propose a re-parametrisation of the alpha-divergence objectives, deriving a simple inference technique which, together with dropout, can be easily implemented with existing models by simply changing the loss of the model. We demonstrate improved uncertainty estimates and accuracy compared to VI in dropout networks. We study our model's epistemic uncertainty far away from the data using adversarial images, showing that these can be distinguished from non-adversarial images by examining our model's uncertainty.

Motivation & Objective

• Motivate the need for better uncertainty estimation in Bayesian neural networks (BNNs).
• Propose a practical alpha-divergence based inference that works with standard dropout and existing architectures.
• Demonstrate improved uncertainty as well as predictive accuracy over standard dropout VI across tasks.
• Assess epistemic uncertainty far from data and its relation to adversarial examples.

Proposed method

• Reformulate BB-α energy to enable dropout-based approximate inference without changing model architectures.
• Use a reparameterisation with cavity distributions to derive a tractable objective compatible with dropout (equation 7).
• Define the MC objective L̃α(q) as KL[q||p0] + const − (1/α) sum_n log-sum-exp[−α l(y_n, f^ω_k(x_n))] with K samples.
• Specialise to dropout by sampling multiple stochastic forward passes, yielding a practical loss (equation 9 for classification, equation 10 for regression).
• Provide a concrete dropout-BB-α objective that raises outputs to the power α and averages over MC samples.
• Show that α controls the trade-off between predictive likelihood optimization (α≈1) and variational free energy (α→0).

Experimental results

Research questions

• RQ1 Does alpha-divergence based dropout inference yield better calibrated uncertainty than standard dropout VI?
• RQ2 How do different α values affect regression and classification performance on benchmark datasets?
• RQ3 Can dropout-BB-α improve robustness and detect adversarial inputs via epistemic uncertainty?
• RQ4 What are the practical training-time implications compared to VI and other Bayesian methods?
• RQ5 How does the approach generalize to CNNs and larger architectures?

Key findings

• Non-VI α values (e.g., α=0.5 or α=1) improve predictive log-likelihood and often maintain competitive RMSE compared to VI in regression.
• In MNIST classification, α=0.5 (Hellinger value) yields best test RMSE and matches EP value in log-likelihood for fully connected networks; VI (α=0) underperforms on these metrics.
• For CNNs on MNIST, VI-α (α=0) can perform comparably to α=0.5 and often near α=1 in log-likelihood, with improvements in accuracy.
• The approach enables MC dropout with a simple loss reformulation, and training time is competitive with VI.
• Uncertainty increases for adversarial MNIST images, enabling separation from non-adversarial samples via epistemic uncertainty.
• The experiments show the method outperforms a Gaussian VI baseline and is competitive with HMC and sparse GP in regression tasks.