[Paper Review] Semantic Segmentation using Adversarial Networks

Pauline Luc, Camille Couprie+2 more|arXiv (Cornell University)|Nov 25, 2016

Anomaly Detection Techniques and ApplicationsComputer Science485 citations

TL;DR

The paper introduces adversarial training for semantic segmentation by training a segmentation CNN jointly with an adversarial network that discriminates ground-truth label maps from predicted ones, improving accuracy on Stanford Background and PASCAL VOC 2012.

ABSTRACT

Adversarial training has been shown to produce state of the art results for generative image modeling. In this paper we propose an adversarial training approach to train semantic segmentation models. We train a convolutional semantic segmentation network along with an adversarial network that discriminates segmentation maps coming either from the ground truth or from the segmentation network. The motivation for our approach is that it can detect and correct higher-order inconsistencies between ground truth segmentation maps and the ones produced by the segmentation net. Our experiments show that our adversarial training approach leads to improved accuracy on the Stanford Background and PASCAL VOC 2012 datasets.

Motivation & Objective

• Motivate the use of adversarial training to enforce higher-order consistency in segmentation maps beyond per-pixel losses.
• Develop a framework combining standard cross-entropy loss with an adversarial loss to train segmentation models.
• Show that the adversarial approach improves segmentation accuracy on benchmark datasets.

Proposed method

• Use a hybrid loss: multi-class cross-entropy for per-pixel labels plus an adversarial loss that judges whether a label map is ground-truth or segmentation-produced.
• Train a segmentation CNN to maximize the difficulty of the adversarial network in distinguishing predicted vs ground-truth maps.
• Adversarial model is a CNN that can take either the label map alone or the label map conditioned on RGB input, with architectures including LargeFOV and SmallFOV variants.
• Experiment with different input encodings for the adversarial network (Basic, Product, Scaling) and two field-of-view settings (LargeFOV, SmallFOV).
• Adopt an alternating training scheme where the segmentation network and the adversarial network are updated in turn to stabilize training.

Experimental results

Research questions

• RQ1 Does adversarial training improve semantic segmentation accuracy over standard cross-entropy training on benchmark datasets?
• RQ2 Can a learned adversarial loss capture higher-order dependencies and spatial contiguity beyond pixel-wise losses?
• RQ3 What architectures and input encodings for the adversarial network yield the best performance trade-offs?
• RQ4 How does adversarial training affect overfitting and boundary delineation in segmentation outputs?

Key findings

• Adversarial training yields consistent gains in mean IoU and related metrics on Stanford Background and PASCAL VOC 2012 compared to a cross-entropy baseline.
• The LargeFOV adversarial architecture provides the most effective improvements among the tested variants.
• Different adversarial input encodings (Basic, Product, Scaling) achieve comparable performance, with scaling providing robust results under various settings.
• Adversarial training reduces overfitting and improves spatial contiguity and boundary sharpness in segmentation maps.
• On PASCAL VOC 2012, selected adversarial setups show small but consistent gains over the baseline on validation and competitive results on the test set.