[Paper Review] Quality of Uncertainty Quantification for Bayesian Neural Network Inference

Jiayu Yao, Weiwei Pan+2 more|arXiv (Cornell University)|Jun 24, 2019

Gaussian Processes and Bayesian InferenceComputer Science19 references74 citations

TL;DR

The paper empirically compares 10 inference methods for Bayesian neural networks to assess how well they quantify uncertainty, showing that common metrics like test log-likelihood can mislead and that richer posterior-capturing methods do not always yield better posterior approximations.

ABSTRACT

Bayesian Neural Networks (BNNs) place priors over the parameters in a neural network. Inference in BNNs, however, is difficult; all inference methods for BNNs are approximate. In this work, we empirically compare the quality of predictive uncertainty estimates for 10 common inference methods on both regression and classification tasks. Our experiments demonstrate that commonly used metrics (e.g. test log-likelihood) can be misleading. Our experiments also indicate that inference innovations designed to capture structure in the posterior do not necessarily produce high quality posterior approximations.

Motivation & Objective

• Motivate robust evaluation of uncertainty in Bayesian neural networks beyond standard predictive metrics.
• Compare a broad range of approximate inference methods on regression and classification tasks.
• Investigate how well different methods approximate the true posterior and how that relates to predictive uncertainty.
• Provide guidance on when common uncertainty metrics are reliable or misleading.

Proposed method

• Evaluate 10 inference methods (BBB, PBP, BB-ALPHA, MNF, MVG, BBH, Dropout, Ensemble, SGLD, SGHMC) against ground-truth HMC.
• Create synthetic, ground-truth-like datasets where posterior predictive uncertainty can be meaningfully assessed.
• Use fixed priors and neural networks (1 hidden layer for regression, 2 hidden layers for classification) and optimize with Adam (except HMC/SGLD/SGHMC).
• Assess posterior predictive quality via multiple metrics including RMSE, test marginal log-likelihood (LogLL), Prediction Interval Coverage Probability (PICP), and Mean Prediction Interval Width (MPIW).
• Argue that log-likelihood and calibration metrics can be poor proxies for posterior fidelity and illustrate with ground-truth-like experiments.

Experimental results

Research questions

• RQ1 How do different Bayesian neural network inference methods compare in terms of predictive uncertainty quality?
• RQ2 Do common uncertainty metrics reliably reflect fidelity to the true posterior across tasks and data regimes?
• RQ3 Does incorporating posterior structure via advanced variational families or ensembles translate to better posterior approximations in practice?

Key findings

• Test log-likelihood and calibration metrics can be misleading indicators of posterior fidelity; they may not reflect true posterior approximation quality.
• Some methods that capture posterior structure do not consistently yield better approximations of the ground-truth posterior.
• SGHMC tends to produce posterior predictives most similar to HMC, while SGLD often underestimates uncertainty.
• Ensembles can give unreliable uncertainty estimates if model diversity is not appropriately encouraged.
• Methods with richer divergences or structured variational families do not universally outperform simpler approaches in these experiments.
• Across tasks, many approximate methods produce predictive distributions that understate uncertainty in regions undersampled by data.