[Paper Review] Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

Balaji Lakshminarayanan, Alexander Pritzel+1 more|arXiv (Cornell University)|Dec 5, 2016

Adversarial Robustness in Machine Learning2,980 citations

TL;DR

The paper proposes a simple, scalable non-Bayesian approach using deep ensembles (with optional adversarial training) trained via proper scoring rules to estimate predictive uncertainty, showing calibrated uncertainties and robustness to domain shift across tasks including ImageNet.

ABSTRACT

Deep neural networks (NNs) are powerful black box predictors that have recently achieved impressive performance on a wide spectrum of tasks. Quantifying predictive uncertainty in NNs is a challenging and yet unsolved problem. Bayesian NNs, which learn a distribution over weights, are currently the state-of-the-art for estimating predictive uncertainty; however these require significant modifications to the training procedure and are computationally expensive compared to standard (non-Bayesian) NNs. We propose an alternative to Bayesian NNs that is simple to implement, readily parallelizable, requires very little hyperparameter tuning, and yields high quality predictive uncertainty estimates. Through a series of experiments on classification and regression benchmarks, we demonstrate that our method produces well-calibrated uncertainty estimates which are as good or better than approximate Bayesian NNs. To assess robustness to dataset shift, we evaluate the predictive uncertainty on test examples from known and unknown distributions, and show that our method is able to express higher uncertainty on out-of-distribution examples. We demonstrate the scalability of our method by evaluating predictive uncertainty estimates on ImageNet.

Motivation & Objective

• Motivate the need for reliable predictive uncertainty in deep neural networks beyond standard accuracy.
• Propose a simple, scalable method combining probabilistic NNs, proper scoring rules, adversarial training, and ensembles.
• Evaluate calibration and out-of-distribution robustness across classification and regression benchmarks, including ImageNet.
• Provide a practical, distributed-training-friendly baseline for predictive uncertainty that rivals approximate Bayesian methods.

Proposed method

• Train probabilistic NNs that model p_theta(y|x) using proper scoring rules (e.g., log-likelihood, Brier score).
• For regression, have networks output a mean and variance and minimize the (negative) log-likelihood with a positive variance via a softplus transformation.
• Optionally apply adversarial training to smooth predictive distributions by augmenting with adversarial examples.
• Train an ensemble of M networks (no bagging; use full data per network) and combine predictions by averaging to form a mixture model.
• Approximate the ensemble predictive distribution as Gaussian by matching the mixture mean and variance to enable efficient computation of predictive probabilities and intervals.

Experimental results

Research questions

• RQ1 Can deep ensembles provide well-calibrated predictive uncertainty comparable to or better than approximate Bayesian methods?
• RQ2 Does adversarial training improve the smoothness and reliability of predictive uncertainty?
• RQ3 How does ensemble size affect calibration, uncertainty, and robustness to out-of-distribution data?
• RQ4 Do these non-Bayesian ensembles scale to large datasets (e.g., ImageNet) while maintaining uncertainty quality?
• RQ5 How does the method perform on known vs unknown class distributions (out-of-distribution) in classification tasks?

Key findings

• Deep ensembles with proper scoring-rule training yield well-calibrated predictive uncertainty often comparable to or better than MC-dropout and PBP on regression and classification tasks.
• Adversarial training can further smooth predictive distributions and improve uncertainty estimates on some datasets, though its benefits vary by task and ensemble size.
• Increasing the ensemble size M substantially improves both accuracy and the quality of predictive uncertainty (lower NLL and Brier scores, better calibration).
• On ImageNet, ensembles improve both classification accuracy and predictive uncertainty (lower NLL and Brier score) compared to single models.
• The method yields higher predictive uncertainty for out-of-distribution inputs (e.g., MNIST vs NotMNIST, SVHN vs CIFAR-10), addressing overconfidence on unseen classes.
• When evaluating accuracy at a fixed confidence threshold, deep ensembles are more robust than MC-dropout, reducing overconfident incorrect predictions.