[Paper Review] Towards Out-Of-Distribution Generalization: A Survey

Jiashuo Liu, Zheyan Shen+5 more|arXiv (Cornell University)|Aug 31, 2021

Domain Adaptation and Few-Shot Learning233 references234 citations

TL;DR

This survey formally defines OOD generalization, categorizes methods across the learning pipeline, and surveys datasets and future directions for robust models under distribution shifts.

ABSTRACT

Traditional machine learning paradigms are based on the assumption that both training and test data follow the same statistical pattern, which is mathematically referred to as Independent and Identically Distributed ($i.i.d.$). However, in real-world applications, this $i.i.d.$ assumption often fails to hold due to unforeseen distributional shifts, leading to considerable degradation in model performance upon deployment. This observed discrepancy indicates the significance of investigating the Out-of-Distribution (OOD) generalization problem. OOD generalization is an emerging topic of machine learning research that focuses on complex scenarios wherein the distributions of the test data differ from those of the training data. This paper represents the first comprehensive, systematic review of OOD generalization, encompassing a spectrum of aspects from problem definition, methodological development, and evaluation procedures, to the implications and future directions of the field. Our discussion begins with a precise, formal characterization of the OOD generalization problem. Following that, we categorize existing methodologies into three segments: unsupervised representation learning, supervised model learning, and optimization, according to their positions within the overarching learning process. We provide an in-depth discussion on representative methodologies for each category, further elucidating the theoretical links between them. Subsequently, we outline the prevailing benchmark datasets employed in OOD generalization studies. To conclude, we overview the existing body of work in this domain and suggest potential avenues for future research on OOD generalization. A summary of the OOD generalization methodologies surveyed in this paper can be accessed at http://out-of-distribution-generalization.com.

Motivation & Objective

• Formally characterize the Out-of-Distribution generalization problem and its relation to i.i.d. learning.
• Categorize OOD methods by their position in the learning pipeline: unsupervised representation learning, supervised model learning, and optimization.
• Discuss connections to related topics such as domain adaptation, domain generalization, federated learning, and OOD detection.
• Survey representative methodologies and benchmarks to guide future OOD generalization research.

Proposed method

• Define the OOD problem as P_tr(X,Y) ≠ P_te(X,Y) and unknown during training.
• Classify methods into three pipeline-based categories: unsupervised representation learning, supervised model learning, and optimization.
• Discuss representative approaches in each category, including invariance, causal learning, stable learning, and distributionally robust optimization.
• Outline theoretical connections among methods through causality and invariance perspectives.
• Review benchmark datasets and evaluation considerations for distribution-shift scenarios.

Experimental results

Research questions

• RQ1 What formal definitions best capture the Out-of-Distribution generalization problem and its relation to i.i.d. learning?
• RQ2 How can methods be categorized by their role in the learning pipeline to address distribution shifts?
• RQ3 What are the core techniques within each category (unsupervised, supervised, optimization) that improve OOD generalization?
• RQ4 How are domain adaptation, domain generalization, and related topics connected to OOD generalization?
• RQ5 What benchmarks and metrics best evaluate OOD generalization performance?

Key findings

• The survey provides a formal problem definition and clarifies covariate vs. concept shifts.
• It organizes OOD methods into unsupervised representation learning, supervised model learning, and optimization, with detailed subcategories.
• Invariant learning and causal discovery are highlighted as central strategies for achieving distributional robustness.
• It discusses the role of environment labels, heterogeneity, and intervention-based perspectives in improving OOD generalization.
• Benchmark datasets and evaluation frameworks for OOD generalization are summarized to guide future research.