[Paper Review] Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networks

Qingsong Lv, Ming Ding+8 more|arXiv (Cornell University)|Dec 30, 2021

Advanced Graph Neural NetworksComputer Science36 references52 citations

TL;DR

This paper systematically reproduces 12 prominent heterogeneous GNNs, shows simple homogeneous GNNs can match or outperform them with proper settings, and introduces the Heterogeneous Graph Benchmark (HGB) and a strong Simple-HGN baseline.

ABSTRACT

Heterogeneous graph neural networks (HGNNs) have been blossoming in recent years, but the unique data processing and evaluation setups used by each work obstruct a full understanding of their advancements. In this work, we present a systematical reproduction of 12 recent HGNNs by using their official codes, datasets, settings, and hyperparameters, revealing surprising findings about the progress of HGNNs. We find that the simple homogeneous GNNs, e.g., GCN and GAT, are largely underestimated due to improper settings. GAT with proper inputs can generally match or outperform all existing HGNNs across various scenarios. To facilitate robust and reproducible HGNN research, we construct the Heterogeneous Graph Benchmark (HGB), consisting of 11 diverse datasets with three tasks. HGB standardizes the process of heterogeneous graph data splits, feature processing, and performance evaluation. Finally, we introduce a simple but very strong baseline Simple-HGN--which significantly outperforms all previous models on HGB--to accelerate the advancement of HGNNs in the future.

Motivation & Objective

• Assess how much progress has been made in heterogeneous graph neural networks (HGNNs) by reproducing prominent models under their official codes, datasets, and settings.
• Identify biases and issues in experimental setups that may inflate perceived HGNN progress.
• Provide a standardized benchmark (HGB) to enable fair, reproducible evaluation of HGNNs across diverse tasks.
• Propose a simple, strong baseline (Simple-HGN) to accelerate and clarify HGNN development.

Proposed method

• Systematic reproduction of 12 popular HGNN models using their official code, datasets, and hyperparameters to compare fairly with homogeneous baselines (GCN/GAT).
• Analysis of experimental setups to identify data leakage, improper baselines, and inefficiencies in previous HGNN work.
• Construction of the Heterogeneous Graph Benchmark (HGB) with 11 datasets and three tasks, standardizing data splits, feature processing, and evaluation pipelines.
• Evaluation of HGNNs against standardized baselines and reporting of reproducible performance across tasks.
• Development of Simple-HGN, a GAT-based model augmented with learnable edge-type embeddings, residual connections, and output normalization to demonstrate a strong, simple baseline.

Experimental results

Research questions

• RQ1 Do homogeneous GNNs like GCN and GAT underperform due to suboptimal settings when compared fairly to HGNNs?
• RQ2 What are the main factors contributing to inconsistent or inflated progress claims in HGNN literature?
• RQ3 Can a standardized benchmark (HGB) enable fair, robust comparisons and drive genuine progress in HGNNs?
• RQ4 Does a simple, well-tuned baseline (Simple-HGN) outperform existing HGNN models across diverse heterogeneous graph tasks?

Key findings

• Fair reproduction shows simple GNNs often match or beat HGNNs when inputs and settings are properly handled.
• Some reported results in prior HGNN work arise from data leakage or inappropriate training/testing splits.
• Meta-paths are not always necessary for strong performance on many heterogeneous datasets.
• A standardized benchmark (HGB) and its leaderboard enable reproducible evaluation across tasks and datasets.
• Simple-HGN, a GAT-backed model with edge-type embeddings, residual connections, and output normalization, consistently outperforms previous HGNNs on HGB.