[Paper Review] ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation

Tuan-Hung Vu, Himalaya Jain+3 more|arXiv (Cornell University)|Nov 30, 2018

Domain Adaptation and Few-Shot LearningComputer Science60 references1,403 citations

TL;DR

The paper introduces entropy-based unsupervised domain adaptation for semantic segmentation, presenting direct entropy minimization and an adversarial entropy minimization (AdvEnt) approach, achieving state-of-the-art results on synthetic-to-real benchmarks GTA5→Cityscapes and SYNTHIA→Cityscapes.

ABSTRACT

Semantic segmentation is a key problem for many computer vision tasks. While approaches based on convolutional neural networks constantly break new records on different benchmarks, generalizing well to diverse testing environments remains a major challenge. In numerous real world applications, there is indeed a large gap between data distributions in train and test domains, which results in severe performance loss at run-time. In this work, we address the task of unsupervised domain adaptation in semantic segmentation with losses based on the entropy of the pixel-wise predictions. To this end, we propose two novel, complementary methods using (i) entropy loss and (ii) adversarial loss respectively. We demonstrate state-of-the-art performance in semantic segmentation on two challenging "synthetic-2-real" set-ups and show that the approach can also be used for detection.

Motivation & Objective

• Address the domain gap between synthetic (source) and real (target) data in semantic segmentation.
• Introduce entropy-based objectives to encourage low-entropy (confident) target predictions without target labels.
• Propose an adversarial extension that aligns weighted self-information between source and target domains.
• Explore practical enhancements like training on entropy ranges and incorporating class-ratio priors to boost performance.

Proposed method

• Direct entropy minimization by adding an entropy loss on target predictions to encourage low-entropy outputs (MinEnt).
• A second approach (AdvEnt) uses adversarial training on weighted self-information maps to align source and target prediction structures via a discriminator.
• Define self-information maps I_x as I_x^{(h,w)} = -P_x^{(h,w)} log P_x^{(h,w)} and use a discriminator D to distinguish source vs target; train F to fool D (minimize L_D).
• Combine supervised segmentation loss on source with entropy or adversarial losses on target (L_ent or L_adv) in a unified objective.
• Optionally incorporate class-ratio priors from the source to regularize target predictions (L_cp).
• Discuss connections to self-training and highlight that entropy minimization acts as a soft, range-based proxy for pseudo-labeling without fixed thresholds.

Experimental results

Research questions

• RQ1 Can entropy-based objectives improve domain adaptation for semantic segmentation without target labels?
• RQ2 How does direct entropy minimization (MinEnt) compare to adversarial entropy minimization (AdvEnt) in bridging source-target gaps?
• RQ3 Does incorporating structural information via adversarial alignment of weighted self-information improve results over pixel-wise entropy minimization?
• RQ4 Do entropy-range training and class-ratio priors provide tangible gains in challenging synthetic-to-real setups?

Key findings

• AdvEnt achieves state-of-the-art results on GTA5→Cityscapes and improves over baselines on SYNTHIA→Cityscapes, with notable gains from adversarial structure alignment.
• Direct MinEnt is competitive and often outperforms some baselines, particularly with lower-capacity networks, and benefits from entropy-range training.
• Adversarial alignment of weighted self-information captures structural consistencies and yields additional gains over direct entropy minimization.
• Using ensemble of MinEnt and AdvEnt provides the best overall performance in several setups.
• Incorporating class-ratio priors helps mitigate class biases and improves mIoU in more challenging target-domain layouts.