[Paper Review] Generalizing to unseen domains via distribution matching

Isabela Albuquerque, João Monteiro+3 more|arXiv (Cornell University)|Nov 3, 2019

Domain Adaptation and Few-Shot LearningComputer Science56 references89 citations

TL;DR

G2DM introduces distribution matching across source domains to generalize to unseen domains without access to target data, via adversarial minimization of pairwise H-divergences.

ABSTRACT

Supervised learning results typically rely on assumptions of i.i.d. data. Unfortunately, those assumptions are commonly violated in practice. In this work, we tackle such problem by focusing on domain generalization: a formalization where the data generating process at test time may yield samples from never-before-seen domains (distributions). Our work relies on the following lemma: by minimizing a notion of discrepancy between all pairs from a set of given domains, we also minimize the discrepancy between any pairs of mixtures of domains. Using this result, we derive a generalization bound for our setting. We then show that low risk over unseen domains can be achieved by representing the data in a space where (i) the training distributions are indistinguishable, and (ii) relevant information for the task at hand is preserved. Minimizing the terms in our bound yields an adversarial formulation which estimates and minimizes pairwise discrepancies. We validate our proposed strategy on standard domain generalization benchmarks, outperforming a number of recently introduced methods. Notably, we tackle a real-world application where the underlying data corresponds to multi-channel electroencephalography time series from different subjects, each considered as a distinct domain.

Motivation & Objective

• Formalize domain generalization with a meta-distribution over domains.
• Prove that small pairwise domain divergences imply small divergences between mixtures of domains.
• Derive a generalization bound for unseen domains based on convex hull mixtures.
• Propose an adversarial, tractable algorithm to minimize domain discrepancies using only source data.
• Demonstrate improved out-of-domain performance on standard benchmarks and a real-world EEG application.

Proposed method

• Define a meta-distribution over domains and the convex hull of source domains as mixtures.
• Prove a bound showing reduced H-divergence between mixtures when pairwise source divergences are small.
• Formulate an adversarial training objective that minimizes classifier loss while maximizing domain-discrimination loss, using a three-component network (encoder, classifier, domain discriminators).
• Estimate pairwise H-divergences efficiently with one-vs-all domain discriminators and random projection layers to stabilize training.
• Train with a minimax objective that balances task accuracy and distribution matching without access to test data.

Experimental results

Research questions

• RQ1 Can G2DM outperform ERM under i.i.d. assumptions by using only source-domain information?
• RQ2 How does G2DM compare to existing domain generalization methods across benchmarks like VLCS and PACS?
• RQ3 Does G2DM actually enforce distribution matching across source domains and unseen domains?
• RQ4 What is the impact of test-data access methods during training on performance and stability?
• RQ5 Is G2DM effective when labeling functions shift across domains, as in EEG-based tasks?

Key findings

• G2DM outperforms ERM and several domain generalization baselines on VLCS and PACS benchmarks.
• G2DM reduces estimated H-divergences between source domains and also lowers divergences to unseen domains in most evaluated cases.
• Using source-data-based stopping criteria, G2DM maintains competitive or superior unseen-domain accuracy compared to CIDDG, across multiple access strategies to test data.
• Replacing AlexNet with ResNet-18 improves stability and average performance in PACS experiments.
• In EEG affective state classification, G2DM yields improved accuracy over ERM even when labeling functions differ across subjects.
• The approach demonstrates that reducing pairwise domain discrepancies among sources can generalize to unseen distributions without target-domain data.