[論文レビュー] Towards Out-Of-Distribution Generalization: A Survey

Jiashuo Liu, Zheyan Shen+5 more|arXiv (Cornell University)|Aug 31, 2021

Domain Adaptation and Few-Shot Learning参考文献 233被引用数 234

ひとこと要約

この調査は形式的にOOD一般化を定義し、学習パイプライン全体で方法を分類し、分布シフト下で堅牢なモデルのためのデータセットと今後の方向性を調査する。

ABSTRACT

Traditional machine learning paradigms are based on the assumption that both training and test data follow the same statistical pattern, which is mathematically referred to as Independent and Identically Distributed ($i.i.d.$). However, in real-world applications, this $i.i.d.$ assumption often fails to hold due to unforeseen distributional shifts, leading to considerable degradation in model performance upon deployment. This observed discrepancy indicates the significance of investigating the Out-of-Distribution (OOD) generalization problem. OOD generalization is an emerging topic of machine learning research that focuses on complex scenarios wherein the distributions of the test data differ from those of the training data. This paper represents the first comprehensive, systematic review of OOD generalization, encompassing a spectrum of aspects from problem definition, methodological development, and evaluation procedures, to the implications and future directions of the field. Our discussion begins with a precise, formal characterization of the OOD generalization problem. Following that, we categorize existing methodologies into three segments: unsupervised representation learning, supervised model learning, and optimization, according to their positions within the overarching learning process. We provide an in-depth discussion on representative methodologies for each category, further elucidating the theoretical links between them. Subsequently, we outline the prevailing benchmark datasets employed in OOD generalization studies. To conclude, we overview the existing body of work in this domain and suggest potential avenues for future research on OOD generalization. A summary of the OOD generalization methodologies surveyed in this paper can be accessed at http://out-of-distribution-generalization.com.

研究の動機と目的

• 分布外（Out-of-Distribution、OOD）一般化問題とそのi.i.d.学習との関係を形式的に特徴づける。
• 学習パイプラインにおける位置に基づきOOD手法を分類する：教師なし表現学習、教師ありモデル学習、最適化。
• ドメイン適応、ドメイン一般化、連合学習、OOD検出などの関連トピックとのつながりを検討する。
• 代表的な手法とベンチマークを調査し、今後のOOD一般化研究を導く。

提案手法

• OOD問題を P_tr(X,Y) ≠ P_te(X,Y) かつ学習中には未知であると定義する。
• 方法を三つのパイプラインベースのカテゴリに分類する：教師なし表現学習、教師ありモデル学習、最適化。
• 各カテゴリの代表的アプローチを議論する：不変性、因果学習、安定学習、分布に頑健な最適化を含む。
• 因果性と不変性の視点を通じた手法間の理論的接続を概説する。
• 分布シフトシナリオのベンチマークデータセットと評価の考慮事項をレビューする。

実験結果

リサーチクエスチョン

• RQ1 What formal definitions best capture the Out-of-Distribution generalization problem and its relation to i.i.d. learning?
• RQ2 How can methods be categorized by their role in the learning pipeline to address distribution shifts?
• RQ3 What are the core techniques within each category (unsupervised, supervised, optimization) that improve OOD generalization?
• RQ4 How are domain adaptation, domain generalization, and related topics connected to OOD generalization?
• RQ5 What benchmarks and metrics best evaluate OOD generalization performance?

主な発見

• The survey provides a formal problem definition and clarifies covariate vs. concept shifts.
• It organizes OOD methods into unsupervised representation learning, supervised model learning, and optimization, with detailed subcategories.
• Invariant learning and causal discovery are highlighted as central strategies for achieving distributional robustness.
• It discusses the role of environment labels, heterogeneity, and intervention-based perspectives in improving OOD generalization.
• Benchmark datasets and evaluation frameworks for OOD generalization are summarized to guide future research.