[Paper Review] Evaluating Bayesian Deep Learning Methods for Semantic Segmentation

Jishnu Mukhoti, Yarin Gal|arXiv (Cornell University)|Nov 30, 2018

Anomaly Detection Techniques and Applications62 references150 citations

TL;DR

This paper proposes three uncertainty-evaluation metrics for Bayesian deep learning in semantic segmentation, implements Bayesian DeepLab variants with MC dropout and Concrete dropout on Cityscapes, and benchmarks their uncertainty quality against a deterministic baseline.

ABSTRACT

Deep learning has been revolutionary for computer vision and semantic segmentation in particular, with Bayesian Deep Learning (BDL) used to obtain uncertainty maps from deep models when predicting semantic classes. This information is critical when using semantic segmentation for autonomous driving for example. Standard semantic segmentation systems have well-established evaluation metrics. However, with BDL's rising popularity in computer vision we require new metrics to evaluate whether a BDL method produces better uncertainty estimates than another method. In this work we propose three such metrics to evaluate BDL models designed specifically for the task of semantic segmentation. We modify DeepLab-v3+, one of the state-of-the-art deep neural networks, and create its Bayesian counterpart using MC dropout and Concrete dropout as inference techniques. We then compare and test these two inference techniques on the well-known Cityscapes dataset using our suggested metrics. Our results provide new benchmarks for researchers to compare and evaluate their improved uncertainty quantification in pursuit of safer semantic segmentation.

Motivation & Objective

• Motivate the need for specialized uncertainty metrics in semantic segmentation when applying Bayesian deep learning.
• Introduce three metrics to evaluate pixel-level uncertainty in segmentation outputs.
• Develop probabilistic segmentation models (Bayesian DeepLab) using MC dropout and Concrete dropout.
• Benchmark uncertainty quality of Bayesian methods against a deterministic baseline on Cityscapes.

Proposed method

• Base architecture is DeepLab-v3+ with Xception backbone and ASPP.
• Incorporate dropout-based approximate inference to create Bayesian DeepLab variants: MC dropout and Concrete dropout.
• Insert dropout layers in the middle flow (after every 4 Xception modules) with rate 0.5.
• Define and compute uncertainty metrics: predictive entropy and mutual information.
• Propose two uncertainty-based performance metrics across patches: p(accurate|certain) and p(uncertain|inaccurate), and the composite PAvPU.
• Compare Bayesian variants against deterministic DeepLab using Cityscapes validation set.

Experimental results

Research questions

• RQ1 Can new, task-specific metrics effectively evaluate uncertainty quality in Bayesian semantic segmentation?
• RQ2 How do MC dropout and Concrete dropout compare in producing reliable pixel-wise uncertainty maps for segmentation?
• RQ3 Do Bayesian DeepLab models outperform deterministic baselines on both segmentation accuracy and uncertainty diagnostics on Cityscapes?

Key findings

• Concrete dropout outperforms MC dropout across pixel accuracy, mean accuracy, and mean IOU.
• Both Bayesian DeepLab variants surpass the deterministic baseline on segmentation metrics, with Concrete dropout achieving the best results (IOU 79.12 vs 79.14 for DeepLab-v3+).
• Deterministic models fail to capture epistemic uncertainty via mutual information, unlike Bayesian variants.
• Predictive entropy captures both aleatoric and epistemic uncertainty, while mutual information isolates epistemic uncertainty.
• Bayesian methods produce interpretable uncertainty maps at pixel level, and uncertainty thresholds affect the proposed metrics.
• Temperature-scaled calibration (ECE/MCE) does not differentiate Bayesian from deterministic models, unlike the proposed metrics.