[Paper Review] Generalizing to Unseen Domains via Adversarial Data Augmentation

Riccardo Volpi, Hongseok Namkoong+4 more|arXiv (Cornell University)|May 30, 2018

Domain Adaptation and Few-Shot Learning28 references365 citations

TL;DR

The paper introduces adversarial data augmentation in semantic space using a Wasserstein-based worst-case formulation to improve generalization to unseen domains, with an iterative augmentation procedure and ensemble models for unknown target shifts.

ABSTRACT

We are concerned with learning models that generalize well to different \emph{unseen} domains. We consider a worst-case formulation over data distributions that are near the source domain in the feature space. Only using training data from a single source distribution, we propose an iterative procedure that augments the dataset with examples from a fictitious target domain that is "hard" under the current model. We show that our iterative scheme is an adaptive data augmentation method where we append adversarial examples at each iteration. For softmax losses, we show that our method is a data-dependent regularization scheme that behaves differently from classical regularizers that regularize towards zero (e.g., ridge or lasso). On digit recognition and semantic segmentation tasks, our method learns models improve performance across a range of a priori unknown target domains.

Motivation & Objective

• Motivate learning models that generalize to unseen domains when only a single source distribution is available.
• Formulate a worst-case distributional robustness problem using Wasserstein distance in semantic space.
• Develop an iterative adaptive data augmentation algorithm that adds adversarial samples during training.
• Show that the method acts as data-dependent regularization for softmax losses and enable ensemble strategies across target-distance levels.

Proposed method

• Define a Wasserstein distance D_theta between distributions in the semantic space induced by the network representation.
• Relax the worst-case problem with a penalty parameter gamma to obtain a robust surrogate loss phi_gamma.
• Iteratively perform a maximization step to generate adversarial (hard) samples by solving x^*_gamma = argmax_x { ell(theta; (x, y)) - gamma c_theta((x, y), (x0, y)) }, then a minimization step with SGD on the augmented dataset.
• Interpret the augmented samples as adaptive data augmentation approximating a Tikhonov-regularized Newton step in the semantic space.
• For classification, show that the robust surrogate phi_gamma corresponds to a data-dependent regularizer encouraging alignment between the true-label classifier and a soft average of class-specific classifiers.
• Provide an ensemble approach by training multiple models with different gamma values and selecting at test time using softmax confidences.

Experimental results

Research questions

• RQ1 Can models trained with adversarial data augmentation generalize to unseen target domains without access to target-domain data during training?
• RQ2 How does the semantic-space Wasserstein robustness compare to pixel-space perturbations and standard regularization in improving cross-domain generalization?
• RQ3 What is the role of the ensemble of models with varying target-distance levels in selecting robust predictions at test time?
• RQ4 Do the proposed techniques yield performance gains on both digit recognition and semantic segmentation under diverse unseen domain shifts?

Key findings

• The method yields cross-domain generalization improvements on digit recognition benchmarks across multiple unseen targets.
• The approach improves semantic segmentation performance across different seasons and weather conditions compared to ERM baselines.
• An ensemble of models with varying robustness levels provides further gains when test-time model selection is used based on softmax confidences.
• The maximization step generates adversarial samples that correspond to adaptive, data-dependent Newton-like updates in the semantic space, acting as effective augmentation.
• For softmax losses, the robust surrogate acts as a data-dependent regularizer, aligning the true-label classifier with a weighted average of class classifiers.