[Paper Review] Domain Generalization by Marginal Transfer Learning

Gilles Blanchard, Aniket Anand Deshmukh+3 more|arXiv (Cornell University)|Nov 21, 2017

Domain Adaptation and Few-Shot LearningComputer Science101 references146 citations

TL;DR

The paper reframes domain generalization as supervised learning on an augmented feature space that includes the marginal distribution of features, proposing a universally consistent kernel method analyzed under two data-generative models and validated on synthetic and real data.

ABSTRACT

In the problem of domain generalization (DG), there are labeled training data sets from several related prediction problems, and the goal is to make accurate predictions on future unlabeled data sets that are not known to the learner. This problem arises in several applications where data distributions fluctuate because of environmental, technical, or other sources of variation. We introduce a formal framework for DG, and argue that it can be viewed as a kind of supervised learning problem by augmenting the original feature space with the marginal distribution of feature vectors. While our framework has several connections to conventional analysis of supervised learning algorithms, several unique aspects of DG require new methods of analysis. This work lays the learning theoretic foundations of domain generalization, building on our earlier conference paper where the problem of DG was introduced (Blanchard et al., 2011). We present two formal models of data generation, corresponding notions of risk, and distribution-free generalization error analysis. By focusing our attention on kernel methods, we also provide more quantitative results and a universally consistent algorithm. An efficient implementation is provided for this algorithm, which is experimentally compared to a pooling strategy on one synthetic and three real-world data sets.

Motivation & Objective

• Motivation to generalize predictive models across multiple related but distributionally different tasks without labeled data for new tasks.
• Proposal of a marginal transfer learning framework that augments features with the marginal X-distribution to enable standard supervised learning techniques.
• Development of two statistical generative models for domain generalization and associated risk and generalization analyses.
• Construction of a distribution-free kernel method within the augmented space and demonstration of universal consistency under both models.

Proposed method

• Formulation of domain generalization as learning on augmented inputs (P_X, x) to produce predictions for unseen tasks.
• Definition of two risk notions corresponding to the agnostic generative model (AGM) and the two-stage generative model (2SGM).
• Derivation of distribution-free generalization bounds using a Rademacher/ Campbell-measure framework in the AGM setting.
• Introduction of a kernel machine on the augmented feature space with universal consistency guarantees under both AGM and 2SGM.
• Efficient implementation based on an extension of random Fourier features and analysis of random feature approximation error.
• Empirical evaluation on one synthetic and three real-world datasets, with comparisons to a pooling baseline, plus freely available code.

Experimental results

Research questions

• RQ1 Can domain generalization be addressed by augmenting the input with the marginal distribution of features to form a standard supervised learning problem?
• RQ2 What are the generalization guarantees and risk bounds for such augmented-space methods under different data-generating assumptions?
• RQ3 Does a universal-consistency kernel-based algorithm exist for DG in both agnostic and two-stage generative models?
• RQ4 How does the proposed method perform relative to pooling strategies on synthetic and real-world domain-generalization tasks?

Key findings

• A marginal transfer learning perspective reframes DG as a supervised problem on an augmented space incorporating the marginal X-distribution.
• The authors establish distribution-free generalization bounds and universal consistency for a kernel-based DG method under both AGM and 2SGM.
• An efficient kernel implementation via random Fourier features is provided and analyzed for approximation error.
• The proposed method is empirically validated against pooling strategies on synthetic and three real-world datasets, demonstrating competitive performance.
• Code implementing the algorithm is publicly available for reproducibility.