[论文解读] Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks

Bálint Mucsányi, Michael Kirchhof+1 more|arXiv (Cornell University)|Feb 29, 2024

Complex Systems and Decision Making被引用 10

一句话总结

本论文在 ImageNet 上对十二种不确定性评估方法在七个实际任务上的综合基准测试，评估信息理论分解和基于 Bregman 的本质不确定性与 epistemic 不确定性 的分离是否在实际中可实现。结论是分离通常未实现，任务特定性能占优。

ABSTRACT

Uncertainty quantification, once a singular task, has evolved into a spectrum of tasks, including abstained prediction, out-of-distribution detection, and aleatoric uncertainty quantification. The latest goal is disentanglement: the construction of multiple estimators that are each tailored to one and only one source of uncertainty. This paper presents the first benchmark of uncertainty disentanglement. We reimplement and evaluate a comprehensive range of uncertainty estimators, from Bayesian over evidential to deterministic ones, across a diverse range of uncertainty tasks on ImageNet. We find that, despite recent theoretical endeavors, no existing approach provides pairs of disentangled uncertainty estimators in practice. We further find that specialized uncertainty tasks are harder than predictive uncertainty tasks, where we observe saturating performance. Our results provide both practical advice for which uncertainty estimators to use for which specific task, and reveal opportunities for future research toward task-centric and disentangled uncertainties. All our reimplementations and Weights & Biases logs are available at https://github.com/bmucsanyi/untangle.

研究动机与目标

• Motivate the need to disentangle predictive uncertainty into task-specific components (aleatoric vs. epistemic).
• Assess whether current uncertainty estimators truly disentangle these components in large-scale practice.
• Characterize which estimators perform best for different practical tasks (ID/OOD, abstention, etc.).
• Provide guidance for task-centric development of uncertainty estimators and reproducible benchmarking.

提出的方法

• Reimplement twelve uncertainty quantification estimators in plug-and-play modules and evaluate them on ImageNet-1k across seven tasks.
• Classify estimators into distributional (producing q(f) over class probabilities) and deterministic (scalar uncertainty outputs).
• Apply two disentanglement paradigms (information-theoretical IT and Bregman decompositions) to obtain paired estimators for aleatoric and epistemic components.
• Aggregate distributional outputs via eight aggregation strategies to obtain scalar uncertainty where needed.
• Replicate experiments on CIFAR-10 for robustness checks and dataset-size effects.

Figure 1 : Six out of seven distributional methods exhibit a severely high rank correlation between the information-theoretical aleatoric and epistemic components when evaluated on ImageNet-ReaL. These methods violate a necessary condition of uncertainty disentanglement.

实验结果

研究问题

• RQ1 Do current uncertainty estimators yield truly disentangled aleatoric and epistemic components in practice?
• RQ2 Which estimators are best for specific practical tasks (e.g., OOD detection, abstention, correctness prediction, etc.)?
• RQ3 How do IT and Bregman decompositions behave across large-scale vs. small-scale datasets?
• RQ4 Do conclusions about disentanglement transfer across datasets and tasks?

主要发现

• Disentanglement is generally not achieved in practice; most IT and Bregman decompositions yield highly correlated aleatoric and epistemic estimates across seven distributional methods on ImageNet-ReaL.
• ODD detection benefits from specialized density-based methods (e.g., Mahalanobis) but these do not translate to other tasks like aleatoric or predictive uncertainty.
• Aleatoric uncertainty remains hard to quantify across methods; deep ensembles and dropout show relatively better alignment with ground-truth human uncertainty (ReaL) than many modern density-based methods.
• Correctness prediction performance is saturated across methods; dropout and deep ensembles provide robust results, while specialized OOD detectors (e.g., Mahalanobis) do not generalize well to in-distribution correctness tasks.
• Across tasks, no single method dominates; task-centric recommendations are necessary, with dropout and deep ensembles offering good general-purpose baselines when costs are allowed.

Figure 2 : Mahalanobis—a direct OOD detector, dropout, and shallow ensembles distinguish ID and OOD samples considerably better (AUROC $\geq 0.728$ ) than the baseline (AUROC $=0.674$ ). OOD samples are perturbed by ImageNet-C corruptions of severity level two.