[Paper Review] Pitfalls of Graph Neural Network Evaluation

Oleksandr Shchur, Maximilian Mumme+2 more|arXiv (Cornell University)|Nov 14, 2018

Advanced Graph Neural Networks19 references632 citations

TL;DR

The paper shows that fair comparison of GNN architectures requires standardized training, hyperparameter tuning, and multiple data splits; otherwise, simple models can outperform complex ones depending on evaluation setup.

ABSTRACT

Semi-supervised node classification in graphs is a fundamental problem in graph mining, and the recently proposed graph neural networks (GNNs) have achieved unparalleled results on this task. Due to their massive success, GNNs have attracted a lot of attention, and many novel architectures have been put forward. In this paper we show that existing evaluation strategies for GNN models have serious shortcomings. We show that using the same train/validation/test splits of the same datasets, as well as making significant changes to the training procedure (e.g. early stopping criteria) precludes a fair comparison of different architectures. We perform a thorough empirical evaluation of four prominent GNN models and show that considering different splits of the data leads to dramatically different rankings of models. Even more importantly, our findings suggest that simpler GNN architectures are able to outperform the more sophisticated ones if the hyperparameters and the training procedure are tuned fairly for all models.

Motivation & Objective

• Evaluate the reliability of empirical comparisons among GNN architectures under standard and varied training/evaluation setups.
• Assess how different train/validation/test splits affect model rankings across multiple datasets.
• Demonstrate that fair hyperparameter tuning can let simpler models outperform more sophisticated GNNs.

Proposed method

• Implement four GNN architectures (GCN, MoNet, GAT, GraphSAGE) within a unified framework for fair comparison.
• Standardize training procedures across models (optimizer, initialization, early stopping, batch strategy) and perform a comprehensive hyperparameter grid search.
• Evaluate on eight datasets (four well-known and four new) with 100 random splits and 20 initializations per split.
• Report both per-dataset results and relative performance metrics to assess robustness to data splits.

Experimental results

Research questions

• RQ1 How do different train/validation/test splits influence the relative performance of GNN architectures?
• RQ2 Does fair hyperparameter tuning across models change which architecture appears superior?
• RQ3 Do simple models (e.g., GCN) outperform complex models when evaluation is standardized and averaged over multiple splits?
• RQ4 What is the impact of evaluation setup on reported performance across diverse graph datasets?

Key findings

• GNN-based models generally outperform baselines across datasets, but no single architecture dominates when evaluated fairly.
• GCN often ranks best when performance is normalized by best split, indicating simpler models can rival or outperform more complex ones under fair tuning.
• Model rankings vary dramatically with different data splits, showing results from a single split can be misleading.
• Extremely high variance can occur for some models (e.g., GAT on Amazon datasets) due to weight initialization, affecting average performance.
• Using multiple splits reveals that hyperparameter tuning and training procedure largely drive performance differences rather than architecture alone.