[Paper Review] New Benchmarks for Learning on Non-Homophilous Graphs

Derek Lim, Xiuyu Li+2 more|arXiv (Cornell University)|Apr 3, 2021

Advanced Graph Neural NetworksComputer Science58 references25 citations

TL;DR

The paper introduces larger, more diverse non-homophilous graph datasets and a new homophily measure, then benchmarks simple baselines and non-homophilous GNNs to reveal insights for non-homophily settings.

ABSTRACT

Much data with graph structures satisfy the principle of homophily, meaning that connected nodes tend to be similar with respect to a specific attribute. As such, ubiquitous datasets for graph machine learning tasks have generally been highly homophilous, rewarding methods that leverage homophily as an inductive bias. Recent work has pointed out this particular focus, as new non-homophilous datasets have been introduced and graph representation learning models better suited for low-homophily settings have been developed. However, these datasets are small and poorly suited to truly testing the effectiveness of new methods in non-homophilous settings. We present a series of improved graph datasets with node label relationships that do not satisfy the homophily principle. Along with this, we introduce a new measure of the presence or absence of homophily that is better suited than existing measures in different regimes. We benchmark a range of simple methods and graph neural networks across our proposed datasets, drawing new insights for further research. Data and codes can be found at https://github.com/CUAI/Non-Homophily-Benchmarks.

Motivation & Objective

• Motivate the need for better non-homophilous graph benchmarks beyond small, synthetic datasets.
• Propose larger, real-world non-homophilous datasets with varied contexts and label-topology relationships.
• Introduce a robust homophily measure that mitigates class-imbalance and size effects.
• Reintroduce strong simple methods and evaluate a range of GNNs and baselines on the proposed datasets.
• Provide insights into method performance and scalability in low-homophily regimes.

Proposed method

• Propose a new homophily measure hat{h} that aggregates class-wise neighbor similarity while controlling for class sizes.
• Assemble several large, real-world non-homophilous datasets across domains (e.g., social networks, biology, time-based/citation data) with node features where applicable.
• Benchmark a broad set of methods including MLP, label propagation (1-hop and 2-hop), LINK, SGC, C&S, general GNNs (GCN, GAT, APPNP, JK variants), and non-homophilous GNNs (H2GCN, MixHop, GPR-GNN).
• Evaluate using standard accuracy or ROC-AUC on five fixed train/validation/test splits (ogbn-proteins uses Open Graph Benchmark splits).
• Discuss memory constraints and scalability when applying non-homophilous methods to large datasets.

Experimental results

Research questions

• RQ1 How do existing non-homophilous learning methods perform on larger, real-world non-homophilous graphs?
• RQ2 Does a simple mix of node-feature and topology-based baselines compete with or surpass specialized non-homophilous GNNs across diverse datasets?
• RQ3 How does the newly proposed homophily measure hat{h} relate to traditional edge homophily and to model performance across datasets?
• RQ4 What are the tradeoffs between scalability and performance for non-homophilous graph learning methods?
• RQ5 Can simple two-hop variants improve performance in non-homophilous settings across datasets?

Key findings

• Simple baselines that use either node features or graph topology perform better than random across the proposed datasets.
• Non-homophilous GNNs tend to perform well, though not universally, with two-hop variants often providing gains.
• The traditional low-rank/low-pass assumptions (as in some homophily-driven models) do not hold in non-homophilous settings, reducing the effectiveness of certain simple GNNs.
• LINK, a topology-based baseline, performs well on many datasets despite not using node features.
• Memorial considerations: larger non-homophilous benchmarks reveal memory and scalability challenges for some methods, highlighting a tradeoff between scalability and performance.
• The new hat{h} measure mitigates class-imbalance effects and better captures presence of homophily across diverse datasets.