[Paper Review] Fully Convolutional Adaptation Networks for Semantic Segmentation

Yiheng Zhang, Zhaofan Qiu+3 more|arXiv (Cornell University)|Apr 23, 2018

Domain Adaptation and Few-Shot LearningComputer Science34 references62 citations

TL;DR

Proposes FCAN, a dual-domain adaptation framework for semantic segmentation that combines appearance-level (AAN) and representation-level (RAN) adaptation to bridge gap between synthetic and real scenes; achieves state-of-the-art unsupervised domain adaptation on GTA5→Cityscapes and BDDS.

ABSTRACT

The recent advances in deep neural networks have convincingly demonstrated high capability in learning vision models on large datasets. Nevertheless, collecting expert labeled datasets especially with pixel-level annotations is an extremely expensive process. An appealing alternative is to render synthetic data (e.g., computer games) and generate ground truth automatically. However, simply applying the models learnt on synthetic images may lead to high generalization error on real images due to domain shift. In this paper, we facilitate this issue from the perspectives of both visual appearance-level and representation-level domain adaptation. The former adapts source-domain images to appear as if drawn from the "style" in the target domain and the latter attempts to learn domain-invariant representations. Specifically, we present Fully Convolutional Adaptation Networks (FCAN), a novel deep architecture for semantic segmentation which combines Appearance Adaptation Networks (AAN) and Representation Adaptation Networks (RAN). AAN learns a transformation from one domain to the other in the pixel space and RAN is optimized in an adversarial learning manner to maximally fool the domain discriminator with the learnt source and target representations. Extensive experiments are conducted on the transfer from GTA5 (game videos) to Cityscapes (urban street scenes) on semantic segmentation and our proposal achieves superior results when comparing to state-of-the-art unsupervised adaptation techniques. More remarkably, we obtain a new record: mIoU of 47.5% on BDDS (drive-cam videos) in an unsupervised setting.

Motivation & Objective

• Address domain shift when learning segmentation models from synthetic data by leveraging unlabeled real data.
• Develop a two-pronged adaptation approach combining appearance-level and representation-level invariance.
• Demonstrate improved segmentation performance in unsupervised domain adaptation settings.
• Show that combining AAN and RAN yields state-of-the-art results on GTA5→Cityscapes and BDDS.

Proposed method

• Introduce Appearance Adaptation Networks (AAN) to transfer source-domain images to resemble target-domain style while preserving source content, via content and style losses over CNN feature maps.
• Introduce Representation Adaptation Networks (RAN) with a shared FCN and a domain discriminator to learn domain-invariant representations through adversarial training, extended by an Atrous Spatial Pyramid Pooling (ASPP) for multi-scale context.
• Optimize RAN with a segmentation loss on source data and an adversarial loss to fool the domain discriminator; use multi-scale ASPP to enhance adversarial learning.
• Implement AAN with a white-noise input that is updated by gradient descent to minimize combined content and style distances; set a small alpha to balance content/style.
• Use a pre-trained ResNet-50 for AAN feature extraction; adopt a dilated FCN (ResNet-101 backbone) with ASPP and a four-branch dilated discriminator in RAN.
• Train with SGD in Caffe, pre-train on source with segmentation loss, then fine-tune with adversarial loss and segmentation loss (lambda=5).

Experimental results

Research questions

• RQ1 Can combining appearance-level and representation-level domain adaptation improve unsupervised semantic segmentation across domains?
• RQ2 How much does each component (AAN, RAN, ASPP) contribute to cross-domain segmentation performance?
• RQ3 What are the gains when transferring from GTA5 to Cityscapes and to BDDS, and can multi-scale (MS) extensions further improve results?
• RQ4 Is semi-supervised extension with some labeled target-domain data beneficial over fully unsupervised adaptation?

Key findings

• FCAN achieves 46.60 mIoU on Cityscapes in the unsupervised GTA5→Cityscapes setting (with all components).
• AAN alone improves performance; combining AAN with RAN yields the best mIoU (46.60) and further gains via late fusion (46.60→46.60).
• RAN provides substantial gains through adversarial domain adaptation, with ADA, Conv, and ASPP contributing 5.78%, 1.88%, and 1.64% respectively, totaling a 9.3% uplift when added to FCN.
• MS extension yields 47.75% mIoU on Cityscapes with FCAN(MS).
• On BDDS, FCAN achieves 43.35% mIoU, FCAN(MS) 45.47%, and FCAN(MS+EN) 47.53% with ensemble; these surpass FCNWild by 3.98% to 7.16% depending on setup.
• Semi-supervised adaptation shows further gains when introducing labeled target data, e.g., 50 labeled images yields 56.50 mIoU for FCAN vs. 47.57 for FCN; 1000 images yield 69.17 for FCAN vs. 68.05 for FCN.