[Paper Review] A Fair Comparison of Graph Neural Networks for Graph Classification

Federico Errica, Marco Podda+2 more|arXiv (Cornell University)|Dec 20, 2019

Advanced Graph Neural Networks29 references118 citations

TL;DR

The paper conducts a large, standardized, and reproducible fair evaluation of five GNN architectures for graph classification across nine datasets, including structure-agnostic baselines, to assess true gains from graph topology and the impact of degree features.

ABSTRACT

Experimental reproducibility and replicability are critical topics in machine learning. Authors have often raised concerns about their lack in scientific publications to improve the quality of the field. Recently, the graph representation learning field has attracted the attention of a wide research community, which resulted in a large stream of works. As such, several Graph Neural Network models have been developed to effectively tackle graph classification. However, experimental procedures often lack rigorousness and are hardly reproducible. Motivated by this, we provide an overview of common practices that should be avoided to fairly compare with the state of the art. To counter this troubling trend, we ran more than 47000 experiments in a controlled and uniform framework to re-evaluate five popular models across nine common benchmarks. Moreover, by comparing GNNs with structure-agnostic baselines we provide convincing evidence that, on some datasets, structural information has not been exploited yet. We believe that this work can contribute to the development of the graph learning field, by providing a much needed grounding for rigorous evaluations of graph classification models.

Motivation & Objective

• Highlight reproducibility issues in GNN graph classification studies and establish a standardized evaluation framework.
• Re-evaluate five popular GNN architectures under the same data splits and identical node features.
• Assess how much structural information contributes beyond node features by using structure-agnostic baselines.
• Investigate the impact of including node degree features on performance and model depth for social graphs.
• Provide publicly available code and dataset splits to enable rigorous future comparisons.

Proposed method

• Review common reproducibility pitfalls in graph classification studies and define a rigorous evaluation protocol.
• Use 10-fold cross-validation for model assessment with an inner 90/10 split for model selection.
• Employ identical input features across models and compare against two structure-agnostic baselines.
• Re-implement five GNN models (DGCNN, DiffPool, ECC, GIN, GraphSAGE) in PyTorch Geometric for fair comparison.
• Evaluate on nine datasets (4 chemical, 5 social) and report mean accuracy with standard deviations.
• Release code and data splits to enable replication.

Experimental results

Research questions

• RQ1 To what extent do state-of-the-art GNNs outperform simple structure-agnostic baselines across graph classification benchmarks?
• RQ2 How much of the performance gains come from graph structure versus node features?
• RQ3 Does including node degree as an input feature consistently improve results on social graphs and affect the required model depth?
• RQ4 Are there datasets where current GNNs fail to beat structure-agnostic baselines, indicating underutilization of topology?

Key findings

• On several chemical datasets (D&D, PROTEINS, ENZYMES), structure-agnostic baselines match or outperform GNNs.
• On NCI1, GNNs clearly exploit graph structure by outperforming the baseline.
• In social datasets, adding node degree features generally improves performance, and can reduce the needed number of layers for some models.
• GIN performs strongly on social datasets, while on some chemical datasets the baselines remain competitive.
• Including degree features can substantially boost baseline performance, and can alter the relative ranking of models.
• The study emphasizes the importance of baselines for fair assessment and reproducibility in graph classification.