[Paper Review] Domain Adaptation for Semantic Segmentation with Maximum Squares Loss

Minghao Chen, Hongyang Xue+1 more|arXiv (Cornell University)|Sep 30, 2019

Domain Adaptation and Few-Shot Learning34 references39 citations

TL;DR

This paper introduces the maximum squares loss to address probability imbalance in unsupervised domain adaptation for semantic segmentation, boosting target-domain training so difficult samples receive meaningful gradients without using adversarial discriminators.

ABSTRACT

Deep neural networks for semantic segmentation always require a large number of samples with pixel-level labels, which becomes the major difficulty in their real-world applications. To reduce the labeling cost, unsupervised domain adaptation (UDA) approaches are proposed to transfer knowledge from labeled synthesized datasets to unlabeled real-world datasets. Recently, some semi-supervised learning methods have been applied to UDA and achieved state-of-the-art performance. One of the most popular approaches in semi-supervised learning is the entropy minimization method. However, when applying the entropy minimization to UDA for semantic segmentation, the gradient of the entropy is biased towards samples that are easy to transfer. To balance the gradient of well-classified target samples, we propose the maximum squares loss. Our maximum squares loss prevents the training process being dominated by easy-to-transfer samples in the target domain. Besides, we introduce the image-wise weighting ratio to alleviate the class imbalance in the unlabeled target domain. Both synthetic-to-real and cross-city adaptation experiments demonstrate the effectiveness of our proposed approach. The code is released at https://github. com/ZJULearning/MaxSquareLoss.

Motivation & Objective

• Motivate reducing labeling cost by enabling unsupervised domain adaptation from synthetic to real data in semantic segmentation.
• Identify limitations of entropy minimization in UDA due to gradient bias toward easy-to-transfer samples.
• Propose the maximum squares loss to balance target-domain gradients and improve class-wise alignment.
• Address target-domain class imbalance with an image-wise weighting scheme and multi-level self-produced guidance to boost low-level features.

Proposed method

• Define maximum squares loss as the target loss term: L_T(x_t) = -(1/2N) sum_n sum_c (p_t^{n,c})^2, which yields a linearly growing gradient to balance easy/difficult samples.
• Show that minimizing the maximum squares loss is equivalent to maximizing Pearson chi-squared divergence between target predictions and the uniform distribution, promoting class-wise distribution alignment.
• Introduce an image-wise weighting factor to handle class imbalance in the unlabeled target domain, computed per image based on predicted class presence.
• Incorporate multi-level self-produced guidance by combining high-level and low-level outputs to generate pseudo-guidance for low-level feature learning (self-produced guidance).
• Provide practical training details and hyperparameters, using ResNet-based backbones and Deeplabv2 style architectures, with and without image-level weighting and multi-level guidance.

Figure 1 : In UDA, the gradient of the entropy minimization method ( $H$ ) is focused on well-classified samples in the target domain. Consequently, we propose the maximum squares loss ( $MS$ ), which is the negative sum of squared probabilities. The gradient of the maximum squares loss is linearly

Experimental results

Research questions

• RQ1 Can maximum squares loss address the bias of entropy minimization in UDA for semantic segmentation by balancing gradients across target classes?
• RQ2 Does a per-image class-balanced weighting scheme improve handling of target-domain class imbalance without target labels?
• RQ3 Do multi-level self-produced guidance techniques further improve performance by guiding low-level feature learning using higher-level predictions?

Key findings

• Maximum squares loss balances gradient contributions across target classes and improves training efficiency for difficult samples compared to entropy minimization.
• Optimizing the maximum squares loss corresponds to maximizing Pearson chi-squared divergence to the uniform distribution, promoting class-wise distribution alignment between domains.
• Image-wise class-balanced weighting significantly enhances performance over traditional global class weighting in UDA for semantic segmentation.
• Adding multi-level self-produced guidance yields notable performance gains by leveraging high-level predictions to guide low-level feature learning.
• Across synthetic-to-real (GTA5/SYNTHIA to Cityscapes) and cross-city adaptation, MaxSquare-based methods achieve competitive or state-of-the-art results without requiring discriminators.

Figure 2 : From GTA5 to Cityscapes, the mean of prediction probability v.s. Intersection over Union(IoU) for each target class. They are almost linearly related. Thus well-classified classes (high IoU) have larger prediction probability.