[Paper Review] Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment

Yifan Wu, Ezra Winston+2 more|arXiv (Cornell University)|Mar 5, 2019

Domain Adaptation and Few-Shot Learning23 references58 citations

TL;DR

The paper introduces asymmetrically-relaxed distribution alignment for domain adaptation, replacing exact distribution matching with a beta-admissible distance to remain robust under label-shift, and provides theory and empirical results showing improved target-domain performance.

ABSTRACT

Domain adaptation addresses the common problem when the target distribution generating our test data drifts from the source (training) distribution. While absent assumptions, domain adaptation is impossible, strict conditions, e.g. covariate or label shift, enable principled algorithms. Recently-proposed domain-adversarial approaches consist of aligning source and target encodings, often motivating this approach as minimizing two (of three) terms in a theoretical bound on target error. Unfortunately, this minimization can cause arbitrary increases in the third term, e.g. they can break down under shifting label distributions. We propose asymmetrically-relaxed distribution alignment, a new approach that overcomes some limitations of standard domain-adversarial algorithms. Moreover, we characterize precise assumptions under which our algorithm is theoretically principled and demonstrate empirical benefits on both synthetic and real datasets.

Motivation & Objective

• Motivate domain adaptation under distribution shifts where supports may not overlap and label shift may occur.
• Propose a relaxed, asymmetric alignment objective that upper-bounds density ratios between target and source representations.
• Provide theoretical guarantees for target-domain performance under concrete, verifiable assumptions.
• Develop beta-admissible distances (f-divergences, modified Wasserstein, reweighting) compatible with adversarial training.
• Demonstrate empirical improvements over standard domain-adversarial methods on synthetic and real data.

Proposed method

• Replace exact distribution matching with beta-admissible distances in adversarial domain adaptation.
• Define D_beta to satisfy zero distance when sup_z p_T^phi(z)/p_S^phi(z) <= 1+beta.
• Present three practical instantiations: beta-admissible f-divergences, modified Wasserstein distance, and reweighting distances.
• Derive adversarial training objectives that realize these beta-admissible distances (including dual form adaptations).
• Propose an implicit-reweighting-by-sorting scheme to optimize reweighting distances within mini-batch training.
• Provide a theoretical bound on target error: E_T(phi,h) <= (1+beta) E_S(phi,h) + 3 delta1 + 2(1+beta) delta2 + delta3 under specific Conditions.

Experimental results

Research questions

• RQ1 Can standard domain-adversarial objectives fail under label distribution shift due to exact distribution matching?
• RQ2 Does asymmetrically-relaxed distribution alignment mitigate failures and provide target-domain guarantees under label-shift assumptions?
• RQ3 How can beta-admissible distances be constructed and optimized in adversarial training to enforce relaxed alignment?
• RQ4 Do beta-admissible approaches improve target accuracy on synthetic data with label shift and perform comparably when shifts are absent?

Key findings

• Under label distribution shift, vanilla DANN fails to maintain target accuracy due to cross-label mappings in latent space.
• beta-admissible distances enable successful adaptation, improving target accuracy on synthetic data (e.g., from 89% to 99% in the reported setup).
• Theoretical bound shows target error is controlled by a scaled source error plus small additive terms under clearly stated conditions.
• Three practical beta-admissible distances are developed and can be optimized via adversarial training.
• Empirical results on synthetic and real datasets show improved target performance with asymmetrically-relaxed alignment and competitive performance otherwise.