[Paper Review] OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs

Weihua Hu, Matthias Fey+4 more|arXiv (Cornell University)|Mar 17, 2021

Advanced Graph Neural Networks62 references120 citations

TL;DR

OGB-LSC introduces three real-world, ultra-large graph datasets (MAG240M, WikiKG90M, PCQM4M) for node-, link-, and graph-level prediction, along with scalable baselines and KDD Cup 2021 outcomes showing expressive GNNs outperform simple baselines at scale.

ABSTRACT

Enabling effective and efficient machine learning (ML) over large-scale graph data (e.g., graphs with billions of edges) can have a great impact on both industrial and scientific applications. However, existing efforts to advance large-scale graph ML have been largely limited by the lack of a suitable public benchmark. Here we present OGB Large-Scale Challenge (OGB-LSC), a collection of three real-world datasets for facilitating the advancements in large-scale graph ML. The OGB-LSC datasets are orders of magnitude larger than existing ones, covering three core graph learning tasks -- link prediction, graph regression, and node classification. Furthermore, we provide dedicated baseline experiments, scaling up expressive graph ML models to the massive datasets. We show that expressive models significantly outperform simple scalable baselines, indicating an opportunity for dedicated efforts to further improve graph ML at scale. Moreover, OGB-LSC datasets were deployed at ACM KDD Cup 2021 and attracted more than 500 team registrations globally, during which significant performance improvements were made by a variety of innovative techniques. We summarize the common techniques used by the winning solutions and highlight the current best practices in large-scale graph ML. Finally, we describe how we have updated the datasets after the KDD Cup to further facilitate research advances. The OGB-LSC datasets, baseline code, and all the information about the KDD Cup are available at https://ogb.stanford.edu/docs/lsc/ .

Motivation & Objective

• Provide large-scale, realistic graph ML benchmarks across node-, link-, and graph-level tasks.
• Evaluate the performance gap between simple scalable baselines and expressive GNNs at scale.
• Facilitate community engagement through benchmarks, baselines, and leaderboards for large-scale graph learning.
• Analyze practical insights from a major KDD Cup 2021 competition.
• Offer dataset updates to sustain progress in large-scale graph ML.

Proposed method

• Introduce three datasets (MAG240M, WikiKG90M, PCQM4M) with node/edge/graph statistics.
• Provide RoBERTa-based textual features for entities/papers and standard graph features.
• Benchmark a range of models from simple baselines (MLP, Label Propagation) to scalable expressive GNNs (GraphSAGE, GAT) with neighbor sampling.
• Extend to heterogeneous graphs with relation-specific weights (R-GraphSAGE, R-GAT) in MAG240M.
• Use standard evaluation metrics: accuracy for node classification, MRR for KG completion, MAE for graph regression.
• Report KDD Cup 2021 results and summarize common techniques used by winners.

Experimental results

Research questions

• RQ1 How do large-scale graphs affect the performance of expressive GNNs versus simple baselines?
• RQ2 What is the impact of heterogeneity and relation-specific modeling on node-level prediction in MAG240M?
• RQ3 Can knowledge graph completion benefit from combined textual and structural encoders on WikiKG90M?
• RQ4 What are the achievable performance gains for graph-level molecular property prediction on PCQM4M with large, deep GNNs?
• RQ5 What lessons from KDD Cup 2021 advance best practices for scaling graph ML?

Key findings

• Expressive GNNs with neighbor sampling significantly outperform simple baselines on MAG240M and WikiKG90M, with gains up to several percentage points in accuracy or MRR.
• Utilizing heterogeneous relations in MAG240M yields notable improvements over homogeneous settings.
• Combining textual information with structural embeddings (Concatenation encoders) provides strong performance on WikiKG90M for KG completion.
• Deep, large GNNs with global message passing and, in some cases, 3D molecular structure information substantially improve PCQM4M/PCQM4Mv2 results, approaching chemistry-relevant benchmarks.
• KDD Cup 2021 winners achieve substantial improvements over baselines, highlighting the value of model ensembling, temporal information, and advanced architectures (e.g., Transformer-based UniMP, KG encoders).
• WikiKG90Mv2 introduces a harder negative selection setup, reflecting realistic KG completion challenges and reducing optimistic scores.