[Paper Review] GOOD: A Graph Out-of-Distribution Benchmark

Shurui Gui, Xiner Li+2 more|arXiv (Cornell University)|Jun 16, 2022

Artificial Intelligence in Healthcare30 citations

TL;DR

GOOD develops a systematic graph OOD benchmark with covariate and concept shifts across 11 datasets (graph and node tasks), enabling 51 domain splits and evaluation of 10 baselines, revealing significant ID–OOD gaps.

ABSTRACT

Out-of-distribution (OOD) learning deals with scenarios in which training and test data follow different distributions. Although general OOD problems have been intensively studied in machine learning, graph OOD is only an emerging area of research. Currently, there lacks a systematic benchmark tailored to graph OOD method evaluation. In this work, we aim at developing an OOD benchmark, known as GOOD, for graphs specifically. We explicitly make distinctions between covariate and concept shifts and design data splits that accurately reflect different shifts. We consider both graph and node prediction tasks as there are key differences in designing shifts. Overall, GOOD contains 11 datasets with 17 domain selections. When combined with covariate, concept, and no shifts, we obtain 51 different splits. We provide performance results on 10 commonly used baseline methods with 10 random runs. This results in 510 dataset-model combinations in total. Our results show significant performance gaps between in-distribution and OOD settings. Our results also shed light on different performance trends between covariate and concept shifts by different methods. Our GOOD benchmark is a growing project and expects to expand in both quantity and variety of resources as the area develops. The GOOD benchmark can be accessed via https://github.com/divelab/GOOD/.

Motivation & Objective

• Motivate the need for a graph-specific OOD benchmark that distinguishes covariate and concept shifts.
• Design a systematic, domain-controlled split strategy for graph data that reflects both covariate and concept shifts.
• Provide a diverse suite of graph-level, node-level, and graph regression tasks to broaden graph OOD evaluation.

Proposed method

• Define covariate shifts by partitioning domains via input features (X_ind) independent of Y to create P_train(X) != P_test(X) while P(Y|X) remains constant.
• Define concept shifts by introducing spurious domain–output correlations (P(Y|X_ind)) that change across concepts while ensuring X_inv fully determines Y.
• Design environments (domains and concepts) to support invariant learning frameworks (ICP/IRM) and enable controlled OOD evaluation.
• Assemble 11 datasets (6 graph-level, 5 node-level) with 51 domain splits (covariate, concept, no-shift) and evaluate 10 baseline methods across 10 random runs.
• Provide dataset processing details and scripts, with an emphasis on graph + node prediction tasks and inclusion of graph regression datasets.

Experimental results

Research questions

• RQ1 How do covariate and concept shifts differently impact graph-based learning across varied datasets and tasks?
• RQ2 Can a single benchmark systematically compare covariate and concept shift effects on graph models within the same domain selections?
• RQ3 What are the baseline performance gaps between in-distribution and out-of-distribution settings across graph and node prediction tasks?
• RQ4 Do graph-specific OOD methods show consistent advantages across both graph-level and node-level tasks under covariate and concept shifts?

Key findings

• Significant performance gaps exist between in-distribution and out-of-distribution settings across 51 splits and 11 datasets.
• In many cases, OOD validation sets outperform ID validation in selecting models with better generalization.
• Graph-specific OOD methods show varying effectiveness across covariate vs. concept shifts, highlighting shift-type–dependent trends.
• GOOD provides a broad, diverse benchmark including graph classification, regression, and node classification, enabling robust future method development.