[Paper Review] Adversarial Learning for Semi-Supervised Semantic Segmentation

Wei-Chih Hung, Yi‐Hsuan Tsai+3 more|arXiv (Cornell University)|Feb 22, 2018

Advanced Neural Network ApplicationsComputer Science26 references377 citations

TL;DR

The paper introduces a fully convolutional discriminator within a GAN framework to improve semi-supervised semantic segmentation by aligning predicted probability maps with ground-truth maps and leveraging unlabeled data via self-taught supervisory signals, achieving gains on VOC 2012 and Cityscapes.

ABSTRACT

We propose a method for semi-supervised semantic segmentation using an adversarial network. While most existing discriminators are trained to classify input images as real or fake on the image level, we design a discriminator in a fully convolutional manner to differentiate the predicted probability maps from the ground truth segmentation distribution with the consideration of the spatial resolution. We show that the proposed discriminator can be used to improve semantic segmentation accuracy by coupling the adversarial loss with the standard cross entropy loss of the proposed model. In addition, the fully convolutional discriminator enables semi-supervised learning through discovering the trustworthy regions in predicted results of unlabeled images, thereby providing additional supervisory signals. In contrast to existing methods that utilize weakly-labeled images, our method leverages unlabeled images to enhance the segmentation model. Experimental results on the PASCAL VOC 2012 and Cityscapes datasets demonstrate the effectiveness of the proposed algorithm.

Motivation & Objective

• Motivate reducing annotation burden in semantic segmentation by leveraging unlabeled data.
• Propose an adversarial framework where a segmentation network (generator) is guided by a fully convolutional discriminator.
• Enable semi-supervised learning by deriving trustworthy regions from discriminator outputs to supervise unlabeled data.
• Eliminate post-processing during testing by integrating adversarial loss into end-to-end training.

Proposed method

• Use a segmentation network (DeepLab-v2 with ResNet-101) producing H x W x C probability maps.
• Introduce a fully convolutional discriminator D that takes probability maps and outputs a spatial confidence map H x W x 1 indicating ground-truth vs. segmentation source.
• Train D with spatial cross-entropy to distinguish ground-truth maps from segmentation outputs.
• Train S with a multi-task loss: L_seg = L_ce + lambda_adv L_adv + lambda_semi L_semi.
• L_adv encourages S to produce maps that fool D (maximize D(S(X)) being classified as ground-truth).
• For unlabeled data, compute a confidence map via D, and apply a masked cross-entropy loss L_semi using a self-taught target; threshold T_semi selects trustworthy regions.

Experimental results

Research questions

• RQ1 Can an adversarial discriminator operating on pixel-wise probability maps improve segmentation quality without increasing inference cost?
• RQ2 Can unlabeled images contribute to segmentation performance through discriminator-derived confidence maps and self-taught supervision?
• RQ3 How does semi-supervised training with adversarial signals compare to fully-supervised baselines on standard datasets?
• RQ4 What are the effects of discriminator architecture (fully convolutional) and hyper-parameters on semi-supervised gains?

Key findings

• Adversarial loss provides consistent gains over the baseline across different labeled data amounts on VOC 2012 (1/8 to full).
• Incorporating L_adv yields 1.6% to 2.8% improvements; combining with L_semi adds 3.5% to 4.0% improvements on VOC 2012.
• On Cityscapes, L_adv provides 0.5% to 1.9% gains, and combining with L_semi yields 1.6% to 3.3% improvements.
• The fully convolutional discriminator is essential, with non-FCD setups underperforming by about 1% (VOC) to 1.5% (overall ablations).
• Semi-supervised learning relies on discriminator-derived confidence maps to mask the training signal; effective with T_semi in [0.1, 0.3].
• Compared to prior adversarial approaches, the proposed method achieves higher mean IU on VOC 2012 under semi-supervised settings.