[Paper Review] Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks

Bálint Mucsányi, Michael Kirchhof+1 more|arXiv (Cornell University)|Feb 29, 2024

Complex Systems and Decision Making10 citations

TL;DR

The paper comprehensively benchmarks twelve uncertainty estimators across seven practical tasks on ImageNet to assess whether information-theoretic and Bregman-based disentanglement of aleatoric and epistemic uncertainty is achievable in practice. It concludes that disentanglement is generally not achieved and that task-specific performance prevails.

ABSTRACT

Uncertainty quantification, once a singular task, has evolved into a spectrum of tasks, including abstained prediction, out-of-distribution detection, and aleatoric uncertainty quantification. The latest goal is disentanglement: the construction of multiple estimators that are each tailored to one and only one source of uncertainty. This paper presents the first benchmark of uncertainty disentanglement. We reimplement and evaluate a comprehensive range of uncertainty estimators, from Bayesian over evidential to deterministic ones, across a diverse range of uncertainty tasks on ImageNet. We find that, despite recent theoretical endeavors, no existing approach provides pairs of disentangled uncertainty estimators in practice. We further find that specialized uncertainty tasks are harder than predictive uncertainty tasks, where we observe saturating performance. Our results provide both practical advice for which uncertainty estimators to use for which specific task, and reveal opportunities for future research toward task-centric and disentangled uncertainties. All our reimplementations and Weights & Biases logs are available at https://github.com/bmucsanyi/untangle.

Motivation & Objective

• Motivate the need to disentangle predictive uncertainty into task-specific components (aleatoric vs. epistemic).
• Assess whether current uncertainty estimators truly disentangle these components in large-scale practice.
• Characterize which estimators perform best for different practical tasks (ID/OOD, abstention, etc.).
• Provide guidance for task-centric development of uncertainty estimators and reproducible benchmarking.

Proposed method

• Reimplement twelve uncertainty quantification estimators in plug-and-play modules and evaluate them on ImageNet-1k across seven tasks.
• Classify estimators into distributional (producing q(f) over class probabilities) and deterministic (scalar uncertainty outputs).
• Apply two disentanglement paradigms (information-theoretical IT and Bregman decompositions) to obtain paired estimators for aleatoric and epistemic components.
• Aggregate distributional outputs via eight aggregation strategies to obtain scalar uncertainty where needed.
• Replicate experiments on CIFAR-10 for robustness checks and dataset-size effects.

Figure 1 : Six out of seven distributional methods exhibit a severely high rank correlation between the information-theoretical aleatoric and epistemic components when evaluated on ImageNet-ReaL. These methods violate a necessary condition of uncertainty disentanglement.

Experimental results

Research questions

• RQ1 Do current uncertainty estimators yield truly disentangled aleatoric and epistemic components in practice?
• RQ2 Which estimators are best for specific practical tasks (e.g., OOD detection, abstention, correctness prediction, etc.)?
• RQ3 How do IT and Bregman decompositions behave across large-scale vs. small-scale datasets?
• RQ4 Do conclusions about disentanglement transfer across datasets and tasks?

Key findings

• Disentanglement is generally not achieved in practice; most IT and Bregman decompositions yield highly correlated aleatoric and epistemic estimates across seven distributional methods on ImageNet-ReaL.
• ODD detection benefits from specialized density-based methods (e.g., Mahalanobis) but these do not translate to other tasks like aleatoric or predictive uncertainty.
• Aleatoric uncertainty remains hard to quantify across methods; deep ensembles and dropout show relatively better alignment with ground-truth human uncertainty (ReaL) than many modern density-based methods.
• Correctness prediction performance is saturated across methods; dropout and deep ensembles provide robust results, while specialized OOD detectors (e.g., Mahalanobis) do not generalize well to in-distribution correctness tasks.
• Across tasks, no single method dominates; task-centric recommendations are necessary, with dropout and deep ensembles offering good general-purpose baselines when costs are allowed.

Figure 2 : Mahalanobis—a direct OOD detector, dropout, and shallow ensembles distinguish ID and OOD samples considerably better (AUROC $\geq 0.728$ ) than the baseline (AUROC $=0.674$ ). OOD samples are perturbed by ImageNet-C corruptions of severity level two.