[Paper Review] FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation

Judy Hoffman, Dequan Wang+2 more|arXiv (Cornell University)|Dec 8, 2016

Adversarial Robustness in Machine Learning30 references692 citations

TL;DR

Introduces the first unsupervised domain adaptation method for semantic segmentation with fully convolutional networks, combining global domain adversarial training and category-specific constrained learning to transfer from labeled source domains to unlabeled target domains.

ABSTRACT

Fully convolutional models for dense prediction have proven successful for a wide range of visual tasks. Such models perform well in a supervised setting, but performance can be surprisingly poor under domain shifts that appear mild to a human observer. For example, training on one city and testing on another in a different geographic region and/or weather condition may result in significantly degraded performance due to pixel-level distribution shift. In this paper, we introduce the first domain adaptive semantic segmentation method, proposing an unsupervised adversarial approach to pixel prediction problems. Our method consists of both global and category specific adaptation techniques. Global domain alignment is performed using a novel semantic segmentation network with fully convolutional domain adversarial learning. This initially adapted space then enables category specific adaptation through a generalization of constrained weak learning, with explicit transfer of the spatial layout from the source to the target domains. Our approach outperforms baselines across different settings on multiple large-scale datasets, including adapting across various real city environments, different synthetic sub-domains, from simulated to real environments, and on a novel large-scale dash-cam dataset.

Motivation & Objective

• Motivate semantic segmentation under domain shift and reduce reliance on target annotations.
• Develop a pixel-level domain adaptation framework for FCNs that handles global and category-specific shifts.
• Leverage adversarial training to align source/target representations and transfer spatial layout via constraints.
• Evaluate the approach on synthetic-to-real, cross-season, and cross-city datasets and introduce a new BDDS dataset.

Proposed method

• Global domain alignment via pixel-level domain adversarial training using region-based instances corresponding to receptive fields.
• Alternating optimization of a domain classifier and the feature space to minimize domain discrepancy between source and target representations.
• Category-specific adaptation through a constrained multiple instance learning objective that transfers source scene layout statistics to the target domain.
• Derivation of a joint objective combining supervised source segmentation loss with global domain alignment and constrained MIL loss.
• Use of dilated FCN front-end (based on VGG16) as backbone for segmentation and adaptation.
• Prediction of image-level labels on the target domain to enforce source-derived class size constraints during adaptation.

Experimental results

Research questions

• RQ1 Can unsupervised domain adaptation improve semantic segmentation when target labels are unavailable?
• RQ2 Does combining global domain alignment with category-specific, layout-aware constraints outperform either component alone?
• RQ3 How well does the approach transfer between synthetic and real domains, seasons, and cities in driving scenarios?
• RQ4 What is the impact of transferring spatial layout statistics on segmentation performance across domains?

Key findings

• Global domain alignment (GA) provides notable mIoU gains over non-adapted baselines in large shifts (synthetic-to-real, GTA5/SYNTHIA to Cityscapes).
• Category-specific adaptation (CA) yields additional gains by enforcing source-like spatial layout in the target, especially for certain classes.
• GA + CA outperforms GA-only in several cross-domain settings, showing the value of combining global alignment with layout transfer.
• Across seasons, the method achieves average improvements (~3 percentage points in mIoU) for most class categories.
• Cross-city adaptation within Cityscapes shows substantial gains primarily from domain adversarial training, with further gains for a subset of categories from CA.
• Introduction of BDDS, a large-scale dash-cam dataset demonstrating qualitative improvements after adaptation.