[Paper Review] ValUES: A Framework for Systematic Validation of Uncertainty Estimation in Semantic Segmentation

Kim-Celine Kahl, Carsten T. Lüth+3 more|arXiv (Cornell University)|Jan 16, 2024

Machine Learning and Data Classification4 citations

TL;DR

This paper introduces ValUES, a systematic framework for validating uncertainty estimation in semantic segmentation, addressing critical gaps in current evaluation practices. By enabling controlled study of data ambiguities and distribution shifts, systematic ablation of method components, and benchmarking across five key applications—OoD detection, active learning, failure detection, calibration, and ambiguity modeling—it reveals that uncertainty type separation is fragile in real data, aggregation strategies are crucial but overlooked, and test-time augmentation often outperforms ensembles in efficiency while maintaining strong performance.

ABSTRACT

Uncertainty estimation is an essential and heavily-studied component for the reliable application of semantic segmentation methods. While various studies exist claiming methodological advances on the one hand, and successful application on the other hand, the field is currently hampered by a gap between theory and practice leaving fundamental questions unanswered: Can data-related and model-related uncertainty really be separated in practice? Which components of an uncertainty method are essential for real-world performance? Which uncertainty method works well for which application? In this work, we link this research gap to a lack of systematic and comprehensive evaluation of uncertainty methods. Specifically, we identify three key pitfalls in current literature and present an evaluation framework that bridges the research gap by providing 1) a controlled environment for studying data ambiguities as well as distribution shifts, 2) systematic ablations of relevant method components, and 3) test-beds for the five predominant uncertainty applications: OoD-detection, active learning, failure detection, calibration, and ambiguity modeling. Empirical results on simulated as well as real-world data demonstrate how the proposed framework is able to answer the predominant questions in the field revealing for instance that 1) separation of uncertainty types works on simulated data but does not necessarily translate to real-world data, 2) aggregation of scores is a crucial but currently neglected component of uncertainty methods, 3) While ensembles are performing most robustly across the different downstream tasks and settings, test-time augmentation often constitutes a light-weight alternative. Code is at: https://github.com/IML-DKFZ/values

Motivation & Objective

• To address the growing gap between theoretical claims and practical performance in uncertainty estimation for semantic segmentation.
• To identify and resolve key pitfalls in current literature, including lack of controlled evaluation, insufficient ablation of method components, and narrow validation across downstream tasks.
• To provide a standardized, systematic validation framework that enables reliable comparison and selection of uncertainty methods for real-world applications.
• To empirically answer fundamental questions about uncertainty type separation, component importance, and method performance across diverse use cases.

Proposed method

• The framework introduces a controlled environment to simulate data ambiguities and distribution shifts, enabling isolation of aleatoric and epistemic uncertainty.
• It enables systematic ablation of four core components: segmentation backbone (C0), prediction model (C1), uncertainty measure (C2), and aggregation strategy (C3).
• The framework includes dedicated test-beds for five key downstream tasks: OoD detection, active learning, failure detection, calibration, and ambiguity modeling.
• It uses both synthetic (toy dataset) and real-world data (LIDC-IDRI, GTA5/Cityscapes) to evaluate method behavior under controlled and realistic conditions.
• Uncertainty scores are normalized and evaluated at both pixel and image levels, with image-level aggregation critical for tasks like failure detection.
• Empirical evaluation uses AUROC and other metrics across multiple uncertainty methods (e.g., TTA, MI, EE, SSN) to compare performance under varying conditions.

Experimental results

Research questions

• RQ1 Can data-related (aleatoric) and model-related (epistemic) uncertainty be meaningfully separated in practice, especially on real-world data?
• RQ2 Which components of uncertainty methods—such as aggregation strategy or uncertainty measure—are essential for real-world performance?
• RQ3 How do different uncertainty methods perform across diverse downstream tasks like OoD detection, active learning, and failure detection?
• RQ4 Does test-time augmentation primarily capture aleatoric or epistemic uncertainty, and how does it compare to ensembles?
• RQ5 To what extent does uncertainty method performance generalize across different data distributions and domain shifts?

Key findings

• Separation of aleatoric and epistemic uncertainty works reliably on simulated data but fails to generalize to real-world data, particularly under domain shifts.
• Aggregation of uncertainty scores from pixel-level to image-level is a crucial but often neglected component; poor aggregation strategies degrade performance in image-level tasks like failure detection.
• Ensembles consistently outperform other methods across all downstream tasks and settings, demonstrating robustness, though test-time augmentation offers a strong, lightweight alternative.
• Test-time augmentation (TTA) is more effective at modeling epistemic uncertainty than previously claimed, contrary to earlier assumptions that it captures aleatoric uncertainty.
• The method of uncertainty estimation significantly impacts downstream performance: for example, MI-based uncertainty achieved AUROC scores of 0.94 in image-level failure detection on the LIDC-MAL dataset, outperforming other methods.
• On the GTA5/Cityscapes dataset, image-level uncertainty aggregation improved failure detection AUROC from 0.70 (patch-level) to 0.82, highlighting the importance of proper aggregation.