[Paper Review] Unsupervised Domain Adaptation by Backpropagation

Yaroslav Ganin, Victor Lempitsky|arXiv (Cornell University)|Sep 26, 2014

Domain Adaptation and Few-Shot Learning31 references2,601 citations

TL;DR

Introduces a gradient reversal layer to learn domain-invariant features while training a label predictor, enabling unsupervised domain adaptation within standard backpropagation for deep networks; achieves state-of-the-art results on Office datasets and strong results on digit and synthetic-to-real tasks.

ABSTRACT

Top-performing deep architectures are trained on massive amounts of labeled data. In the absence of labeled data for a certain task, domain adaptation often provides an attractive option given that labeled data of similar nature but from a different domain (e.g. synthetic images) are available. Here, we propose a new approach to domain adaptation in deep architectures that can be trained on large amount of labeled data from the source domain and large amount of unlabeled data from the target domain (no labeled target-domain data is necessary). As the training progresses, the approach promotes the emergence of "deep" features that are (i) discriminative for the main learning task on the source domain and (ii) invariant with respect to the shift between the domains. We show that this adaptation behaviour can be achieved in almost any feed-forward model by augmenting it with few standard layers and a simple new gradient reversal layer. The resulting augmented architecture can be trained using standard backpropagation. Overall, the approach can be implemented with little effort using any of the deep-learning packages. The method performs very well in a series of image classification experiments, achieving adaptation effect in the presence of big domain shifts and outperforming previous state-of-the-art on Office datasets.

Motivation & Objective

• Motivate domain adaptation when target-domain labels are unavailable and leverage abundant labeled source-domain data.
• Embed domain adaptation into deep feature learning to produce discriminative yet domain-invariant representations.
• Develop a simple SGD-compatible training procedure that integrates a gradient reversal layer into existing architectures.

Proposed method

• Propose a three-part network: feature extractor G_f, label predictor G_y, and domain classifier G_d operating on G_f’s output.
• Introduce a gradient reversal layer between G_f and G_d that multiplies the backpropagated domain loss by -λ, encouraging domain-invariant features.
• Formulate a saddle-point objective that minimizes label-loss on source data while maximizing domain-loss, controlled by λ.
• Optimize via standard SGD-style updates with a backward pass through the gradient reversal layer, yielding a domain-invariant feature space.
• Relate the approach to HΔH-distance-based generalization bounds and show that the domain classifier’s performance bounds domain discrepancy.

Experimental results

Research questions

• RQ1 Can a deep network be trained end-to-end for unsupervised domain adaptation by jointly optimizing for discriminative labels and domain-invariance?
• RQ2 Does introducing a gradient reversal mechanism effectively align feature distributions across source and target domains during learning?
• RQ3 How does the proposed method perform on standard domain adaptation benchmarks compared to prior approaches?
• RQ4 Can this method be adapted to real-world datasets with synthetic-to-real and cross-domain shifts (e.g., Office dataset)?

Key findings

• Achieves substantial improvement over source-only models across multiple cross-domain tasks.
• On MNIST to MNIST-M, the method reaches 0.8149 accuracy vs. 0.5749 baseline and outperforms the SA baseline.
• On Syn Numbers to SVHN, the method reaches 0.9048 accuracy vs. 0.8665 baseline and outperforms SA.
• On SVHN to MNIST, the method reaches 0.7107 accuracy vs. 0.5919 baseline and outperforms SA.
• On Syn Signs to GTSRB, the method reaches 0.8866 accuracy vs. 0.7400 baseline and outperforms SA.
• Office dataset experiments show the proposed approach achieves 0.673±0.017 (Amazon→DSLR/Webcam) and 0.940±0.008 (DSLR→Webcam) and 0.937±0.010 (Webcam→DSLR), outperforming previous methods.