[Paper Review] ESL: Entropy-guided Self-supervised Learning for Domain Adaptation in Semantic Segmentation

Antoine Saporta, Tuan-Hung Vu+2 more|arXiv (Cornell University)|Jun 15, 2020

Domain Adaptation and Few-Shot Learning26 references33 citations

TL;DR

ESL uses entropy as the confidence criterion for pseudo-labels in unsupervised domain adaptation for semantic segmentation, improving self-training and achieving state-of-the-art results on multiple benchmarks.

ABSTRACT

While fully-supervised deep learning yields good models for urban scene semantic segmentation, these models struggle to generalize to new environments with different lighting or weather conditions for instance. In addition, producing the extensive pixel-level annotations that the task requires comes at a great cost. Unsupervised domain adaptation (UDA) is one approach that tries to address these issues in order to make such systems more scalable. In particular, self-supervised learning (SSL) has recently become an effective strategy for UDA in semantic segmentation. At the core of such methods lies `pseudo-labeling', that is, the practice of assigning high-confident class predictions as pseudo-labels, subsequently used as true labels, for target data. To collect pseudo-labels, previous works often rely on the highest softmax score, which we here argue as an unfavorable confidence measurement. In this work, we propose Entropy-guided Self-supervised Learning (ESL), leveraging entropy as the confidence indicator for producing more accurate pseudo-labels. On different UDA benchmarks, ESL consistently outperforms strong SSL baselines and achieves state-of-the-art results.

Motivation & Objective

• Motivate robust semantic segmentation under domain shift without dense target annotations.
• Leverage entropy as a confidence measure to build higher-quality pseudo-labels for the target domain.
• Integrate entropy-guided pseudo-labeling into an iterative domain-alignment/self-training framework.
• Demonstrate improvements over softmax-based SSL baselines across synthetic-to-real domain adaptation benchmarks.

Proposed method

• Adopt adversarial domain alignment as the core UDA mechanism, with a discriminator D and segmentation model F.
• Train F to minimize a source segmentation loss plus an adversarial loss that aligns target predictions with the source domain.
• Iterate a self-supervised learning step where pseudo-labels are generated on target images from F and used to further train F.
• Replace standard softmax-based pseudo-labeling with entropy-based pseudo-label extraction, filtering target pixels by entropy thresholds per class.
• Define entropy as E = - (1/log C) sum_c P(c) log P(c) and use class-specific entropy thresholds to select reliable pseudo-labels.
• Use thresholds mu*(for softmax) and nu*(for entropy) with dataset-specific heuristic values to balance label quantity and quality.

Experimental results

Research questions

• RQ1 Can entropy serve as a more reliable confidence proxy than maximum softmax probability for selecting target pseudo-labels in SSL-based UDA for semantic segmentation?
• RQ2 Does entropy-guided pseudo-labeling improve the quality of pseudo-labels and translate into measurable gains in target-domain segmentation performance?
• RQ3 How does ESL interact with existing domain-alignment strategies (e.g., adversarial output-space or feature-space alignment) in iterative self-training pipelines?
• RQ4 What are robust guidelines (thresholds) for entropy-based pseudo-label selection across different domain shifts (synthetic-to-real datasets)?

Key findings

• Entropy-based pseudo-labeling consistently improves performance over strong SSL baselines across multiple UDA benchmarks.
• ESL yields state-of-the-art results in the tested settings, outperforming softmax-based pseudo-labeling in mean IoU metrics.
• The entropy criterion more effectively filters boundary and uncertain regions, reducing confirmation bias from incorrect pseudo-labels.
• A single iteration of ESL (with entropy-guided pseudo-labeling) provides notable gains when combined with existing adversarial domain alignment methods.
• Entropy-based pseudo-labeling reduces incorrect pseudo-labels on easier classes, contributing to cleaner self-training signals.