[Paper Review] Bridging Theory and Algorithm for Domain Adaptation

Yuchen Zhang, Tianle Liu+2 more|arXiv (Cornell University)|Apr 11, 2019

Domain Adaptation and Few-Shot Learning46 references229 citations

TL;DR

This paper extends domain adaptation theory to multiclass scoring functions with margin loss by introducing Margin Disparity Discrepancy (MDD) and derives generalization bounds; it then turns the theory into an adversarial learning algorithm that achieves state-of-the-art results on standard benchmarks.

ABSTRACT

This paper addresses the problem of unsupervised domain adaption from theoretical and algorithmic perspectives. Existing domain adaptation theories naturally imply minimax optimization algorithms, which connect well with the domain adaptation methods based on adversarial learning. However, several disconnections still exist and form the gap between theory and algorithm. We extend previous theories (Mansour et al., 2009c; Ben-David et al., 2010) to multiclass classification in domain adaptation, where classifiers based on the scoring functions and margin loss are standard choices in algorithm design. We introduce Margin Disparity Discrepancy, a novel measurement with rigorous generalization bounds, tailored to the distribution comparison with the asymmetric margin loss, and to the minimax optimization for easier training. Our theory can be seamlessly transformed into an adversarial learning algorithm for domain adaptation, successfully bridging the gap between theory and algorithm. A series of empirical studies show that our algorithm achieves the state of the art accuracies on challenging domain adaptation tasks.

Motivation & Objective

• Extend domain adaptation theory to multiclass classification with scoring functions and margin loss.
• Introduce Margin Disparity Discrepancy (MDD) with margin-aware generalization bounds.
• Provide a practical adversarial learning algorithm that minimizes MDD for domain adaptation.
• Demonstrate state-of-the-art empirical performance on standard benchmarks.

Proposed method

• Define margin-based discrepancy (MDD) that upper-bounds target error via source margin error and distribution discrepancy.
• Develop generalization bounds using Rademacher complexity and covering numbers for MDD in multiclass domain adaptation.
• Formulate a minimax optimization where a feature extractor and classifier minimize empirical margin loss and MDD, with an auxiliary classifier maximizing the margin-discrepancy.
• Implement an adversarial network with a gradient reversal layer to optimize the representation for reduced MDD.
• Use a combined cross-entropy loss to approximate margin-based objectives for practical optimization.
• Provide margin parameter gamma (exp(rho)) controls the margin-based discrepancy and relates to the equilibrium of the adversarial game.

Experimental results

Research questions

• RQ1 How can margin loss be integrated into domain adaptation theory for multiclass problems?
• RQ2 Can a margin-based discrepancy measure (MDD) provide tighter, trainable generalization bounds than traditional divergences in domain adaptation?
• RQ3 Does an adversarial algorithm built around MDD achieve state-of-the-art accuracy on standard domain adaptation benchmarks?
• RQ4 What is the trade-off between margin level (rho) and generalization/optimization in practice?

Key findings

• MDD achieves state-of-the-art accuracy on Office-31, outperforming several baselines across multiple transfer tasks.
• On Office-31, MDD reaches an average accuracy of 88.9% versus competing methods.
• On Office-Home, MDD attains strong gains over prior methods (as reported in Table 2).
• On VisDA-2017, MDD achieves 74.6% accuracy (Synthetic→Real), higher than JAN, MCD, GTA, and CDAN baselines.
• The proposed margin-based theory yields practical adversarial learning algorithms with competitive to superior empirical performance across diverse domain shifts.