[Paper Review] A critical look at the evaluation of GNNs under heterophily: Are we really making progress?

Oleg Platonov, Denis Kuznedelev+3 more|arXiv (Cornell University)|Feb 22, 2023

Tracheal and airway disorders22 citations

TL;DR

This paper reveals data leakage and duplication issues in standard heterophily benchmarks, proposes diverse new heterophilous datasets, and finds that standard GNNs often outperform heterophily-specific models.

ABSTRACT

Node classification is a classical graph machine learning task on which Graph Neural Networks (GNNs) have recently achieved strong results. However, it is often believed that standard GNNs only work well for homophilous graphs, i.e., graphs where edges tend to connect nodes of the same class. Graphs without this property are called heterophilous, and it is typically assumed that specialized methods are required to achieve strong performance on such graphs. In this work, we challenge this assumption. First, we show that the standard datasets used for evaluating heterophily-specific models have serious drawbacks, making results obtained by using them unreliable. The most significant of these drawbacks is the presence of a large number of duplicate nodes in the datasets Squirrel and Chameleon, which leads to train-test data leakage. We show that removing duplicate nodes strongly affects GNN performance on these datasets. Then, we propose a set of heterophilous graphs of varying properties that we believe can serve as a better benchmark for evaluating the performance of GNNs under heterophily. We show that standard GNNs achieve strong results on these heterophilous graphs, almost always outperforming specialized models. Our datasets and the code for reproducing our experiments are available at https://github.com/yandex-research/heterophilous-graphs

Motivation & Objective

• Highlight flaws in widely used heterophily benchmarks (e.g., duplicate nodes and train-test leakage).
• Propose diverse, large-scale heterophilous datasets as a better benchmark for evaluating GNNs under heterophily.
• Empirically compare standard GNNs and heterophily-specific models on the new benchmark to assess progress.

Proposed method

• Critically analyze existing heterophily datasets and measure their properties (adjusted homophily, label informativeness).
• Identify duplicates causing train-test leakage and quantify performance shifts after removal.
• Create five new heterogeneous graphs spanning domains with 10K–50K nodes and diverse structural properties.
• Benchmark a broad set of models (standard GNNs and heterophily-specific) on the new datasets.
• Investigate ego- and neighbor-embedding separation as a factor contributing to performance under heterophily.

Experimental results

Research questions

• RQ1 Do standard heterophily benchmarks suffer from data leakage due to duplicates, and how does this affect model rankings?
• RQ2 Do standard GNNs outperform heterophily-specific models on diverse, new heterophilous benchmarks?
• RQ3 Does separating ego- and neighbor-embeddings consistently improve performance under heterophily across datasets?

Key findings

• Duplicate nodes in squirrel and chameleon create train-test leakage; removing duplicates drastically changes model performance.
• On the new heterophilous benchmark, standard GNNs mostly outperform heterophily-specific models across datasets.
• The trick of separating ego- and neighbor-embeddings (as in GAT-sep, GT-sep, etc.) yields strong results for standard GNNs under heterophily.
• FSGNN remains a relatively strong heterophily-specific model but does not consistently dominate; overall, many heterophily-specific methods do not exceed standard GNNs on the proposed benchmarks.
• Best results among baselines typically come from models that separate ego- and neighbor-embeddings, suggesting this is a robust tactic for heterophily settings.