[Paper Review] Wasserstein Distance Guided Representation Learning for Domain Adaptation

Jian Shen, Yanru Qu+2 more|arXiv (Cornell University)|Jul 5, 2017

Domain Adaptation and Few-Shot LearningComputer Science298 citations

TL;DR

WDGRL learns domain-invariant, discriminative representations by adversarially minimizing the Wasserstein distance between source and target features via a domain critic, improving cross-domain performance on sentiment and image classification tasks.

ABSTRACT

Domain adaptation aims at generalizing a high-performance learner on a target domain via utilizing the knowledge distilled from a source domain which has a different but related data distribution. One solution to domain adaptation is to learn domain invariant feature representations while the learned representations should also be discriminative in prediction. To learn such representations, domain adaptation frameworks usually include a domain invariant representation learning approach to measure and reduce the domain discrepancy, as well as a discriminator for classification. Inspired by Wasserstein GAN, in this paper we propose a novel approach to learn domain invariant feature representations, namely Wasserstein Distance Guided Representation Learning (WDGRL). WDGRL utilizes a neural network, denoted by the domain critic, to estimate empirical Wasserstein distance between the source and target samples and optimizes the feature extractor network to minimize the estimated Wasserstein distance in an adversarial manner. The theoretical advantages of Wasserstein distance for domain adaptation lie in its gradient property and promising generalization bound. Empirical studies on common sentiment and image classification adaptation datasets demonstrate that our proposed WDGRL outperforms the state-of-the-art domain invariant representation learning approaches.

Motivation & Objective

• Motivate domain adaptation under covariate shift where target labeled data are scarce.
• Learn feature representations that are invariant to domain while remaining discriminative.
• Leverage Wasserstein distance for stable gradients and better generalization guarantees.
• Integrate WDGRL with a discriminative classifier to ensure target-domain predictivity.

Proposed method

• Introduce a feature extractor f_g that maps inputs to a latent representation.
• Train a domain critic f_w to estimate the Wasserstein distance between source and target representations.
• Enforce a Lipschitz constraint on the critic via gradient penalty and optimize Wasserstein distance with gradient ascent.
• Update the feature extractor to minimize the Wasserstein distance while also minimizing source-domain classification loss.
• Optionally combine WDGRL with a discriminator to ensure discriminative, label-aware representations for the target domain.
• Provide a practical training algorithm that alternates critic maximization and feature/discriminator minimization.

Experimental results

Research questions

• RQ1 Can Wasserstein distance provide reliable gradients for aligning source and target representations in domain adaptation?
• RQ2 Does WDGRL outperform existing domain-invariant representation learning methods (e.g., DANN, MMD, CORAL) on standard benchmarks?
• RQ3 Do the learned representations generalize well to target domains, supported by a theoretical bound?
• RQ4 How does incorporating a supervised discriminator affect target-domain performance and representation quality?

Key findings

• WDGRL outperforms state-of-the-art representation learning approaches on common domain adaptation benchmarks in sentiment and image classification.
• The learned representations are domain-invariant yet discriminative, as evidenced by visualization showing unified domain distributions with preserved label structure.
• Theoretical analysis indicates gradient superiority of Wasserstein distance over traditional adversarial losses, aiding stable training and generalization.
• Empirical results on Amazon review and Office-Caltech datasets demonstrate improved average accuracy across transfer tasks.
• WDGRL can be integrated into existing symmetric feature-based adaptation frameworks by replacing the representation learning component.