[Paper Review] TUDataset: A collection of benchmark datasets for learning with graphs

Morris, Christopher, Kriege, Nils M.+4 more|arXiv (Cornell University)|Jul 16, 2020

Advanced Graph Neural NetworksComputer Science74 references302 citations

TL;DR

The paper introduces TUDataset, a collection of over 120 graph datasets for supervised learning, with standardized evaluation procedures, Python data loaders, and baseline kernel and GNN implementations; it also reports a comparative study of graph kernels and graph neural networks on a subset.

ABSTRACT

Recently, there has been an increasing interest in (supervised) learning with graph data, especially using graph neural networks. However, the development of meaningful benchmark datasets and standardized evaluation procedures is lagging, consequently hindering advancements in this area. To address this, we introduce the TUDataset for graph classification and regression. The collection consists of over 120 datasets of varying sizes from a wide range of applications. We provide Python-based data loaders, kernel and graph neural network baseline implementations, and evaluation tools. Here, we give an overview of the datasets, standardized evaluation procedures, and provide baseline experiments. All datasets are available at www.graphlearning.io. The experiments are fully reproducible from the code available at www.github.com/chrsmrrs/tudataset.

Motivation & Objective

• Provide a large, diverse set of benchmark graph datasets to standardize evaluation in graph learning.
• Enable fair comparisons between graph kernels and graph neural networks (GNNs) across domains.
• Offer Python data loaders, baseline implementations, and reproducible evaluation protocols to the community.
• Assess baseline performance of kernels and GNNs and highlight where traditional kernels remain competitive.

Proposed method

• Assemble and describe a diverse collection of 120+ graph datasets from domains including small molecules, bioinformatics, computer vision, social networks, and synthetic graphs.
• Provide standard dataset formats and Python interfaces compatible with PyTorch Geometric and DGL.
• Implement common graph kernels (e.g., Weisfeiler-Lehman variants, SP, Graphlet) and standard GNN baselines (Gin variants) for baseline comparisons.
• Propose standardized evaluation procedures: cross-validation for kernels via LibSVM/LibLinear, end-to-end optimization for GNNs with Adam, and consistent hyperparameter tuning.
• Conduct an experimental study comparing kernel methods and GNNs on a subset of TUDataset, including large-scale molecular regression tasks (Zinc, QM9, Alchemy).
• Ensure reproducibility with code repositories and documentation (graphlearning.io and GitHub).

Experimental results

Research questions

• RQ1 How do classical graph kernels compare to modern GNNs on a broad, standardized graph classification benchmark?
• RQ2 Which baselines perform best across small, mid, and large-scale graph datasets and what are the scalability trade-offs?
• RQ3 Do standardized evaluation procedures enable fairer, more reproducible comparisons across graph learning methods?
• RQ4 What is the impact of dataset scale (small to large) on the relative performance of kernels versus GNNs?
• RQ5 How can a unified dataset collection facilitate progress in graph representation learning across domains?

Key findings

• Graph kernels (notably WL-OA) often perform competitively on small-scale datasets but do not scale well to large datasets due to Gram matrix computation.
• On larger datasets, the 1-WL kernel and GNN baselines (Gin variants) show competitive performance, with neural methods gaining prominence on certain datasets (e.g., github_stargazers).
• Specialized architectures like MPNN can yield significant gains on large-scale molecular regression tasks over generic GNN baselines.
• For large-scale molecular tasks (Zinc, Alchemy, QM9), explicit molecular architectures (MPNN) outperform baseline GINE-based approaches, highlighting the value of domain-specific models.
• Across the reported results, classical graph kernels combined with SVMs remain highly competitive for graph classification, especially on smaller datasets, while GNNs offer advantages on some mid-to-large scale tasks.