[Paper Review] Benchmarking Graph Neural Networks

Vijay Prakash Dwivedi, Chaitanya K. Joshi+4 more|arXiv (Cornell University)|Mar 2, 2020

Advanced Graph Neural Networks100 references233 citations

TL;DR

The paper presents an open-source, modular benchmarking framework for Graph Neural Networks (GNNs) with a diverse 12-dataset collection, fixed parameter budgets for fair comparisons, and an exploration of graph positional encoding using Laplacian eigenvectors, plus introduction of the AQSOL dataset.

ABSTRACT

In the last few years, graph neural networks (GNNs) have become the standard toolkit for analyzing and learning from data on graphs. This emerging field has witnessed an extensive growth of promising techniques that have been applied with success to computer science, mathematics, biology, physics and chemistry. But for any successful field to become mainstream and reliable, benchmarks must be developed to quantify progress. This led us in March 2020 to release a benchmark framework that i) comprises of a diverse collection of mathematical and real-world graphs, ii) enables fair model comparison with the same parameter budget to identify key architectures, iii) has an open-source, easy-to-use and reproducible code infrastructure, and iv) is flexible for researchers to experiment with new theoretical ideas. As of December 2022, the GitHub repository has reached 2,000 stars and 380 forks, which demonstrates the utility of the proposed open-source framework through the wide usage by the GNN community. In this paper, we present an updated version of our benchmark with a concise presentation of the aforementioned framework characteristics, an additional medium-sized molecular dataset AQSOL, similar to the popular ZINC, but with a real-world measured chemical target, and discuss how this framework can be leveraged to explore new GNN designs and insights. As a proof of value of our benchmark, we study the case of graph positional encoding (PE) in GNNs, which was introduced with this benchmark and has since spurred interest of exploring more powerful PE for Transformers and GNNs in a robust experimental setting.

Motivation & Objective

• Establish a community-standard, fair benchmarking framework for GNNs across diverse real-world and mathematical graphs.
• Provide a modular, reproducible codebase (PyTorch/DGL) to enable fair comparisons under fixed parameter budgets.
• Expand the dataset suite to include essential mathematical graphs and the AQSOL molecular dataset with real-world targets.
• Demonstrate how the framework can drive insights into GNN design, such as graph positional encoding (PE) using Laplacian eigenvectors.

Proposed method

• Introduce a modular GNN benchmarking framework built on PyTorch and DGL with data pipelines, GNN layers/models, training/evaluation, and reproducibility scripts.
• Provide a dataset collection of 12 medium-scale graphs spanning real-world and mathematical domains (Table 1).
• Implement two parameter budgets (100k and 500k) to enable fair comparison of architectures regardless of total parameter count.
• Demonstrate the framework’s use by analyzing graph positional encoding (PE) via Laplacian eigenvectors appended to node features.
• Describe how the framework can be extended to test new ideas in data preprocessing, layers, and normalization schemes.
• Discuss design choices favoring medium-scale datasets for rapid, reliable prototyping.

Experimental results

Research questions

• RQ1 What GNN architectures and building blocks perform best under fixed parameter budgets across diverse graph tasks?
• RQ2 How can graph positional encoding influence the performance and expressivity of GNNs in practical benchmarks?
• RQ3 To what extent does the benchmark distinguish between different GNN categories (MP-GCNs vs WL-GNNs) and datasets across graph-level, node-level, and edge-level tasks?
• RQ4 Can the framework accommodate and accelerate exploration of new GNN ideas, normalization schemes, and pooling mechanisms?

Key findings

• The benchmark framework has been widely used to prototype GNN ideas and study aggregation, expressive power, pooling, normalization, and robustness.
• Graph positional encoding using Laplacian eigenvectors improved MP-GCNs on synthetics and real-world datasets, including the AQSOL dataset.
• The framework facilitated studies that spurred subsequent literature on PE and related GNN enhancements (e.g., Beaini et al., 2021; Wang et al., 2022; Lim et al., 2022; Kreuzer et al., 2021; Ying et al., 2021; Mialon et al., 2021).
• The updated framework adds essential mathematical datasets and the AQSOL molecular dataset to expand evaluation scenarios.
• The GitHub repository achieved community traction (2000+ stars, 380+ forks) and was cited in the literature, illustrating the utility of the open-source infrastructure.