[Paper Review] Open Graph Benchmark: Datasets for Machine Learning on Graphs

Weihua Hu, Matthias Fey+6 more|arXiv (Cornell University)|May 2, 2020

Advanced Graph Neural Networks99 references491 citations

TL;DR

Open Graph Benchmark (OGB) provides a diverse, large-scale suite of graph datasets with standardized splits, evaluation metrics, and an end-to-end pipeline to advance scalable and reproducible graph ML research.

ABSTRACT

We present the Open Graph Benchmark (OGB), a diverse set of challenging and realistic benchmark datasets to facilitate scalable, robust, and reproducible graph machine learning (ML) research. OGB datasets are large-scale, encompass multiple important graph ML tasks, and cover a diverse range of domains, ranging from social and information networks to biological networks, molecular graphs, source code ASTs, and knowledge graphs. For each dataset, we provide a unified evaluation protocol using meaningful application-specific data splits and evaluation metrics. In addition to building the datasets, we also perform extensive benchmark experiments for each dataset. Our experiments suggest that OGB datasets present significant challenges of scalability to large-scale graphs and out-of-distribution generalization under realistic data splits, indicating fruitful opportunities for future research. Finally, OGB provides an automated end-to-end graph ML pipeline that simplifies and standardizes the process of graph data loading, experimental setup, and model evaluation. OGB will be regularly updated and welcomes inputs from the community. OGB datasets as well as data loaders, evaluation scripts, baseline code, and leaderboards are publicly available at https://ogb.stanford.edu .

Motivation & Objective

• Address the lack of scalable, realistic graph ML benchmarks by introducing large-scale, diverse graph datasets across node, link, and graph prediction tasks.
• Provide domain-specific realistic data splits and evaluation metrics to enable fair, reproducible model comparison.
• Offer an automated end-to-end pipeline (data loading, splitting, evaluation) and public leaderboards to accelerate research and benchmarking.

Proposed method

• Introduce a unified benchmark suite (OGB) with datasets spanning Nature, Society, and Information domains.
• Define three graph ML task categories: node property prediction, link property prediction, and graph property prediction.
• Provide domain-specific, realistic data splits and standardized evaluation metrics for each dataset.
• Develop an automated OGB data loader and evaluator within a PyTorch-compatible framework (and support for PyTorch Geometric and DGL).
• Conduct extensive baseline benchmarking using representative GNNs and mini-batch training methods across all datasets.
• Make code, baselines, and leaderboards publicly available and maintain an ongoing community-driven repository.

Experimental results

Research questions

• RQ1 How do large-scale, diverse graph datasets affect the scalability and generalization of graph ML models?
• RQ2 What realistic data splits reveal about out-of-distribution generalization and model robustness across domains?
• RQ3 Can a unified data loading, evaluation, and benchmarking pipeline enable fair, reproducible comparisons for GNNs on real-world graphs?

Key findings

• OGB datasets are large-scale, spanning small to large graphs, and cover multiple domains and tasks.
• Realistic splits (not random) reveal substantial generalization gaps and challenges in out-of-distribution evaluation.
• Mini-batch GNNs can offer competitive or superior performance to full-batch counterparts under memory constraints, with benefits from regularization.
• Baseline results show significant performance gaps between training and test under realistic splits, underscoring generalization challenges.
• A standardized pipeline with data loaders, evaluators, and leaderboards facilitates reproducible benchmarking and progress tracking.