[论文解读] Towards Out-Of-Distribution Generalization: A Survey

Jiashuo Liu, Zheyan Shen+5 more|arXiv (Cornell University)|Aug 31, 2021

Domain Adaptation and Few-Shot Learning参考文献 233被引用 234

一句话总结

本综述正式定义了 OOD 泛化，按学习流程对方法进行分类，并对分布偏移下的鲁棒模型的数据集和未来研究方向进行综述。

ABSTRACT

Traditional machine learning paradigms are based on the assumption that both training and test data follow the same statistical pattern, which is mathematically referred to as Independent and Identically Distributed ($i.i.d.$). However, in real-world applications, this $i.i.d.$ assumption often fails to hold due to unforeseen distributional shifts, leading to considerable degradation in model performance upon deployment. This observed discrepancy indicates the significance of investigating the Out-of-Distribution (OOD) generalization problem. OOD generalization is an emerging topic of machine learning research that focuses on complex scenarios wherein the distributions of the test data differ from those of the training data. This paper represents the first comprehensive, systematic review of OOD generalization, encompassing a spectrum of aspects from problem definition, methodological development, and evaluation procedures, to the implications and future directions of the field. Our discussion begins with a precise, formal characterization of the OOD generalization problem. Following that, we categorize existing methodologies into three segments: unsupervised representation learning, supervised model learning, and optimization, according to their positions within the overarching learning process. We provide an in-depth discussion on representative methodologies for each category, further elucidating the theoretical links between them. Subsequently, we outline the prevailing benchmark datasets employed in OOD generalization studies. To conclude, we overview the existing body of work in this domain and suggest potential avenues for future research on OOD generalization. A summary of the OOD generalization methodologies surveyed in this paper can be accessed at http://out-of-distribution-generalization.com.

研究动机与目标

• 对 Out-of-Distribution 泛化问题及其与独立同分布(i.i.d.)学习的关系进行形式化刻画。
• 按学习流程中的位置对 OOD 方法进行分类：无监督表示学习、监督模型学习和优化。
• 讨论与之相关的主题的联系，如领域自适应、领域泛化、联邦学习及 OOD 检测。
• 综述具有代表性的方法论和基准数据集，以指导未来的 OOD 泛化研究。

提出的方法

• 将 OOD 问题定义为 P_tr(X,Y) ≠ P_te(X,Y)，且在训练期间未知。
• 将方法分类为三种基于流程的类别：无监督表示学习、监督模型学习和优化。
• 讨论每个类别中的代表性方法，包括不变量学习、因果学习、稳健学习以及分布鲁棒优化。
• 从因果性和不变量角度勾勒方法之间的理论联系。
• 评审对于分布偏移场景的基准数据集与评估考虑。

实验结果

研究问题

• RQ1 哪些形式化定义最能捕捉 OOD 泛化问题及其与 i.i.d. 学习的关系？
• RQ2 方法如何按其在学习流程中的角色进行分类以应对分布偏移？
• RQ3 在每个类别（无监督、监督、优化）中有哪些提升 OOD 泛化的核心技术？
• RQ4 领域自适应、领域泛化及相关主题如何与 OOD 泛化相关联？
• RQ5 哪些基准和指标最能评估 OOD 泛化性能？

主要发现

• 本综述提供了正式的问题定义，并澄清了协变量偏移与概念偏移。
• 它将 OOD 方法组织为无监督表示学习、监督模型学习和优化，并给出详细的子类别。
• 不变量学习和因果发现被强调为实现分布鲁棒性的核心策略。
• 它讨论了环境标签、异质性以及基于干预的视角在提升 OOD 泛化中的作用。
• 对 OOD 泛化的基准数据集和评估框架进行了总结，以指导未来的研究。