[Paper Review] Generalizing Across Domains via Cross-Gradient Training

Shiv Shankar, Vihari Piratla+4 more|arXiv (Cornell University)|Apr 28, 2018

Domain Adaptation and Few-Shot LearningComputer Science27 references268 citations

TL;DR

CrossGrad uses domain-guided input perturbations and joint label-domain training to generalize classifiers to unseen domains without target-domain data or explicit domain features. It outperforms domain adversarial and generic perturbation methods across multiple tasks.

ABSTRACT

We present CROSSGRAD, a method to use multi-domain training data to learn a classifier that generalizes to new domains. CROSSGRAD does not need an adaptation phase via labeled or unlabeled data, or domain features in the new domain. Most existing domain adaptation methods attempt to erase domain signals using techniques like domain adversarial training. In contrast, CROSSGRAD is free to use domain signals for predicting labels, if it can prevent overfitting on training domains. We conceptualize the task in a Bayesian setting, in which a sampling step is implemented as data augmentation, based on domain-guided perturbations of input instances. CROSSGRAD parallelly trains a label and a domain classifier on examples perturbed by loss gradients of each other's objectives. This enables us to directly perturb inputs, without separating and re-mixing domain signals while making various distributional assumptions. Empirical evaluation on three different applications where this setting is natural establishes that (1) domain-guided perturbation provides consistently better generalization to unseen domains, compared to generic instance perturbation methods, and that (2) data augmentation is a more stable and accurate method than domain adversarial training.

Motivation & Objective

• Motivate learning from multi-domain data to generalize to unseen domains without target-domain adaptation.
• Propose a domain-guided data augmentation method that perturbs inputs along domain loss gradients.
• Avoid overfitting to training domains while preserving helpful domain signals for in-domain predictions.
• Demonstrate empirical generalization gains across handwriting, font recognition, MNIST rotations, and spoken word tasks.

Proposed method

• Model input x as influenced by label y and domain d via a latent domain feature g.
• Use a domain classifier G to extract continuous domain features g from x and predict d.
• Perturb x in the direction of the domain loss gradient to generate augmented samples with different domain characteristics.
• Train label and domain classifiers with cross-perturbed data to prevent overfitting to training domains.
• Formulate training as alternating updates of two objectives J_l (label) and J_d (domain) with cross-gradients.
• Evaluate CrossGrad against baselines (Baseline, DAN) and LabelGrad on four datasets with disjoint train/test domains.

Experimental results

Research questions

• RQ1 Can CrossGrad generalize to unseen domains without explicit target-domain data or domain features?
• RQ2 Does domain-guided perturbation provide better generalization than generic perturbations or domain-adversarial approaches?
• RQ3 How does CrossGrad perform across diverse tasks (font/handwriting recognition, MNIST rotations, speech commands) and architectures?
• RQ4 In what training conditions (number of domains) is CrossGrad most effective?

Key findings

• CrossGrad improves accuracy over Baseline, DAN, and LabelGrad across all four datasets.
• On Fonts, Handwriting, MNIST, and Speech, CrossGrad achieves accuracies of 72.6, 88.6, 98.6, and 73.5 respectively (vs. Baseline 68.5, 82.5, 95.6, 72.6).
• CrossGrad maintains gains across different architectures (LeNet and ResNet) for Fonts and Handwriting.
• Performance gains from CrossGrad are larger when the number of training domains is small, diminishing as domain coverage increases.
• Domain adversarial networks (DAN) provide unstable gains and are difficult to tune in these settings.
• LabelGrad, while useful, generally yields smaller improvements than CrossGrad, but its effectiveness decreases as domain diversity grows.