[Paper Review] Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures

Benjamin Kompa|arXiv (Cornell University)|Nov 30, 2021

Adversarial Robustness in Machine Learning30 references25 citations

TL;DR

This paper evaluates the empirical frequentist coverage of deep learning uncertainty quantification methods—such as Bayesian neural networks, Monte Carlo dropout, and ensembles—across regression and classification tasks with and without dataset shift. It finds that while some methods achieve good coverage on in-distribution data, coverage degrades significantly under distributional shift, highlighting a critical limitation for real-world deployment.

ABSTRACT

Uncertainty quantification for complex deep learning models is increasingly important as these techniques see growing use in high-stakes, real-world settings. Currently, the quality of a model’s uncertainty is evaluated using point-prediction metrics, such as the negative log-likelihood (NLL), expected calibration error (ECE) or the Brier score on held-out data. Marginal coverage of prediction intervals or sets, a well-known concept in the statistical literature, is an intuitive alternative to these metrics but has yet to be systematically studied for many popular uncertainty quantification techniques for deep learning models. With marginal coverage and the complementary notion of the width of a prediction interval, downstream users of deployed machine learning models can better understand uncertainty quantification both on a global dataset level and on a per-sample basis. In this study, we provide the first large-scale evaluation of the empirical frequentist coverage properties of well-known uncertainty quantification techniques on a suite of regression and classification tasks. We find that, in general, some methods do achieve desirable coverage properties on in distribution samples, but that coverage is not maintained on out-of-distribution data. Our results demonstrate the failings of current uncertainty quantification techniques as dataset shift increases and reinforce coverage as an important metric in developing models for real-world applications.

Motivation & Objective

• To assess the empirical frequentist coverage of popular uncertainty quantification techniques in deep learning.
• To examine how coverage properties change under dataset shift in regression and classification tasks.
• To evaluate the interpretability and utility of coverage and width as metrics for real-world deployment.
• To compare uncertainty methods using coverage, width, Brier score, and ECE across diverse benchmarks.
• To demonstrate that coverage is a more intuitive and reliable metric than traditional calibration scores for downstream users like clinicians.

Proposed method

• Employs marginal coverage as a metric: the fraction of true labels contained within predicted 95% prediction intervals or sets.
• Measures coverage on in-distribution test sets and out-of-distribution corruptions (CIFAR-10-C, ImageNet-C) to assess robustness under dataset shift.
• Uses standard uncertainty quantification techniques: deep ensembles, Monte Carlo dropout, stochastic variational inference (SVI), Gaussian processes, and temperature scaling.
• Computes prediction interval width in standard deviations of the training set (regression) or average set size (classification) to assess calibration efficiency.
• Applies nested cross-validation to tune hyperparameters using validation loss, reflecting real-world practitioner workflows.
• Compares methods using coverage, width, Brier score, and expected calibration error (ECE) to assess consistency across metrics.

Experimental results

Research questions

• RQ1 Do popular uncertainty quantification methods maintain 95% marginal coverage under dataset shift?
• RQ2 How does the width of prediction intervals correlate with dataset shift and coverage performance?
• RQ3 How do coverage rankings compare to traditional metrics like ECE and Brier score?
• RQ4 Which uncertainty methods provide the most robust coverage across diverse benchmarks and shift levels?
• RQ5 Can prediction width serve as a soft proxy for detecting dataset shift in real-world applications?

Key findings

• Ensembles, temperature scaling, and dropout methods maintained higher coverage on ImageNet-C and CIFAR-10-C corruptions compared to SVI and Gaussian processes.
• Coverage degraded significantly under dataset shift, even as prediction interval widths increased, indicating insufficient robustness.
• SVI and LL-SVI achieved the highest coverage with the smallest widths, particularly in regression tasks.
• Prediction set width showed a 0.9 Pearson correlation with translation shift magnitude on MNIST, suggesting width as a proxy for distributional shift.
• Coverage rankings were highly consistent with ECE and Brier score rankings, but coverage was deemed more interpretable for non-technical users.
• No method maintained 95% coverage on out-of-distribution data without a corresponding increase in width that was insufficient to preserve coverage levels.