[Paper Review] Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks

Konstantinos Bousmalis, Nathan Silberman+3 more|arXiv (Cornell University)|Dec 16, 2016

Domain Adaptation and Few-Shot Learning43 references72 citations

TL;DR

This paper proposes PixelDA, a GAN-based unsupervised domain adaptation method that learns to transform synthetic source-domain images to appear as real target-domain images at the pixel level, without requiring paired data. It achieves state-of-the-art performance on object classification and pose estimation tasks, reducing pose estimation error by over 50% in challenging settings and enabling stable, generalizable adaptation across unseen object classes.

ABSTRACT

Collecting well-annotated image datasets to train modern machine learning algorithms is prohibitively expensive for many tasks. One appealing alternative is rendering synthetic data where ground-truth annotations are generated automatically. Unfortunately, models trained purely on rendered images often fail to generalize to real images. To address this shortcoming, prior work introduced unsupervised domain adaptation algorithms that attempt to map representations between the two domains or learn to extract features that are domain-invariant. In this work, we present a new approach that learns, in an unsupervised manner, a transformation in the pixel space from one domain to the other. Our generative adversarial network (GAN)-based method adapts source-domain images to appear as if drawn from the target domain. Our approach not only produces plausible samples, but also outperforms the state-of-the-art on a number of unsupervised domain adaptation scenarios by large margins. Finally, we demonstrate that the adaptation process generalizes to object classes unseen during training.

Motivation & Objective

• To address the domain shift problem where models trained on synthetic data fail to generalize to real images.
• To develop an unsupervised, pixel-level domain adaptation method that does not require labeled target data or paired source-target images.
• To decouple domain adaptation from task-specific models for greater flexibility and reusability.
• To improve training stability and generalization in adversarial domain adaptation via task-specific and content-similarity losses.
• To enable zero-shot generalization to object classes unseen during training.

Proposed method

• A conditional GAN is trained to map source-domain (e.g., synthetic) images to target-domain (e.g., real) images using only unpaired data.
• The generator is trained to minimize adversarial loss against a domain discriminator, while preserving content through a content-similarity loss based on feature reconstruction.
• A task-specific classifier is jointly trained on both real source images and generated target-like images to stabilize training and improve performance.
• The content-similarity loss ensures that the generated images retain the semantic content of the input source image, preventing mode collapse.
• The model is trained end-to-end using a combination of adversarial loss, task-specific classification loss, and content reconstruction loss.
• The adaptation process is decoupled from the task-specific classifier, allowing reuse of the domain adapter across different downstream tasks.

Experimental results

Research questions

• RQ1 Can a GAN-based model effectively perform unsupervised pixel-level domain adaptation without paired source-target image pairs?
• RQ2 Does the proposed method improve performance on downstream tasks like classification and pose estimation compared to state-of-the-art unsupervised domain adaptation techniques?
• RQ3 Can the model generalize to object classes not seen during training, demonstrating zero-shot adaptation?
• RQ4 How does the inclusion of task-specific and content-similarity losses affect training stability and performance variance?
• RQ5 Can the model be effectively used in semi-supervised settings with a small number of labeled target samples?

Key findings

• PixelDA achieves a 99.93% classification accuracy and 13.31° mean angle error on the 'Synthetic Cropped Linemod to Cropped Linemod' benchmark, outperforming prior methods by a large margin.
• In the challenging 'Synthetic Cropped Linemod to Cropped Linemod' scenario, the model reduces pose estimation error by more than 50% compared to the previous state of the art.
• The model generalizes to unseen object classes, achieving 98.98% classification accuracy on 5 previously unseen Linemod objects, demonstrating zero-shot transfer capability.
• The inclusion of task-specific and content-similarity losses reduces performance standard deviation across random initializations from 23.26% to 1.60% for classification accuracy, significantly improving training stability.
• In semi-supervised settings with only 1,000 labeled target samples, PixelDA achieves 99.93% accuracy and 13.31° mean angle error—surpassing baselines trained on both synthetic data and labeled target data.
• The adapted images generated by the model are visually plausible and closely resemble real target-domain images, as shown in qualitative comparisons with real RGBD samples.