[论文解读] GOOD: A Graph Out-of-Distribution Benchmark

Shurui Gui, Xiner Li+2 more|arXiv (Cornell University)|Jun 16, 2022

Artificial Intelligence in Healthcare被引用 30

一句话总结

GOOD 开发了一个系统性图OOD基准，涵盖11个数据集的协变量和概念偏移（图任务和节点任务），实现51个领域切分并评估10个基线，揭示显著的ID–OOD差距。

ABSTRACT

Out-of-distribution (OOD) learning deals with scenarios in which training and test data follow different distributions. Although general OOD problems have been intensively studied in machine learning, graph OOD is only an emerging area of research. Currently, there lacks a systematic benchmark tailored to graph OOD method evaluation. In this work, we aim at developing an OOD benchmark, known as GOOD, for graphs specifically. We explicitly make distinctions between covariate and concept shifts and design data splits that accurately reflect different shifts. We consider both graph and node prediction tasks as there are key differences in designing shifts. Overall, GOOD contains 11 datasets with 17 domain selections. When combined with covariate, concept, and no shifts, we obtain 51 different splits. We provide performance results on 10 commonly used baseline methods with 10 random runs. This results in 510 dataset-model combinations in total. Our results show significant performance gaps between in-distribution and OOD settings. Our results also shed light on different performance trends between covariate and concept shifts by different methods. Our GOOD benchmark is a growing project and expects to expand in both quantity and variety of resources as the area develops. The GOOD benchmark can be accessed via https://github.com/divelab/GOOD/.

研究动机与目标

• 提出需要一个图特定的OOD基准，以区分协变量偏移和概念偏移。
• 设计一个对领域控制的、系统性的分割策略，适用于图数据，能够反映协变量与概念偏移。
• 提供多样化的图级、节点级和图回归任务，拓展图OOD评估。

提出的方法

• 将协变量偏移定义为通过输入特征（X_ind）对领域进行划分，与 Y 无关，产生 P_train(X) != P_test(X) 同时 P(Y|X) 保持不变。
• 将概念偏移定义为引入虚假的领域–输出相关（P(Y|X_ind)）在概念之间改变，同时确保 X_inv 完全决定 Y。
• 设计环境（领域和概念）以支持不变学习框架（ICP/IRM）并实现受控的OOD评估。
• 汇集11个数据集（6个图级，5个节点级）以及51个领域切分（协变量、概念、无偏移），在10次随机运行中评估10个基线方法。
• 提供数据集处理细节和脚本，重点是图+节点预测任务，并包含图回归数据集。

实验结果

研究问题

• RQ1 协变量和概念偏移如何在不同数据集和任务中对基于图的学习产生不同的影响？
• RQ2 是否可以在同一领域选择中，使用单一基准系统地比较协变量和概念偏移对图模型的影响？
• RQ3 在图和节点预测任务中，ID与OOD设置之间的基线性能差距有多大？
• RQ4 在协变量和概念偏移下，图特定的OOD方法在图级和节点级任务上是否表现出一致的优势？

主要发现

• 在51个切分和11个数据集上，ID与OOD设置之间存在显著的性能差距。
• 在许多情况下，OOD验证集在选择具有更好泛化能力的模型时，优于ID验证集。
• 图特定的OOD方法在协变量偏移与概念偏移之间显示出不同的有效性，凸显偏移类型相关的趋势。
• GOOD 提供了一个广泛且多样的基准，涵盖图分类、回归和节点分类，便于未来方法的稳健发展。