[Paper Review] Generalizing to Unseen Domains: A Survey on Domain Generalization

Jindong Wang, Cuiling Lan+7 more|arXiv (Cornell University)|Mar 2, 2021

Domain Adaptation and Few-Shot Learning96 citations

TL;DR

A comprehensive survey of domain generalization (DG): formal definitions, theory, algorithmic taxonomy, datasets, applications, and open research directions, including an open-source codebase DeepDG.

ABSTRACT

Machine learning systems generally assume that the training and testing distributions are the same. To this end, a key requirement is to develop models that can generalize to unseen distributions. Domain generalization (DG), i.e., out-of-distribution generalization, has attracted increasing interests in recent years. Domain generalization deals with a challenging setting where one or several different but related domain(s) are given, and the goal is to learn a model that can generalize to an unseen test domain. Great progress has been made in the area of domain generalization for years. This paper presents the first review of recent advances in this area. First, we provide a formal definition of domain generalization and discuss several related fields. We then thoroughly review the theories related to domain generalization and carefully analyze the theory behind generalization. We categorize recent algorithms into three classes: data manipulation, representation learning, and learning strategy, and present several popular algorithms in detail for each category. Third, we introduce the commonly used datasets, applications, and our open-sourced codebase for fair evaluation. Finally, we summarize existing literature and present some potential research topics for the future.

Motivation & Objective

• Define domain generalization and distinguish it from related fields.
• Survey theoretical foundations and generalization bounds relevant to DG.
• Categorize and analyze major DG algorithms by data manipulation, representation learning, and learning strategies.
• Discuss commonly used datasets, applications, and provide an open-source evaluation resource DeepDG.
• Outline future research directions and potential challenges in DG.

Proposed method

• Formulate DG formally with training source domains and unseen target domains.
• Review and relate DG theory to domain adaptation through bounds like HΔH-divergence and covariate shift analyses.
• Provide a taxonomy of DG methods: data manipulation, representation learning, and learning strategies, with subcategories.
• Discuss domain-invariant representations, feature disentanglement, and causality-inspired approaches.
• Introduce DeepDG, an open-source codebase for DG evaluation and experiments.

Figure 1: Examples from the dataset PACS [ 2 ] for domain generalization. The training set is composed of images belonging to domains of sketch, cartoon, and art paintings. DG aims to learn a generalized model that performs well on the unseen target domain of photos.

Experimental results

Research questions

• RQ1 What formal definitions and relationships with related fields (e.g., domain adaptation, transfer learning) characterize DG?
• RQ2 What theoretical bounds explain DG generalization and how are they estimated in practice?
• RQ3 How can DG methods be categorized and what are the representative algorithms in each category?
• RQ4 What datasets and evaluation protocols are used for DG, and how can fair evaluation be facilitated?
• RQ5 What are promising future directions and open challenges in DG?

Key findings

• DG is formalized as learning from multiple labeled source domains to generalize to unseen target domains with different distributions.
• Theoretical DG bounds leverage concepts like HΔH-divergence and covariate shift, guiding domain-invariant representations.
• DG methods are categorized into data manipulation, representation learning, and learning strategy, each with multiple influential techniques.
• Domain-invariant representations and feature disentanglement are central themes in achieving DG, with various kernel, adversarial, and regularization approaches.
• An open-source DeepDG codebase is provided to support fair evaluation across DG benchmarks and datasets.

Figure 2: Illustration of domain generalization. Adapted from [ 6 ] .