[论文解读] A critical look at the evaluation of GNNs under heterophily: Are we really making progress?

Oleg Platonov, Denis Kuznedelev+3 more|arXiv (Cornell University)|Feb 22, 2023

Tracheal and airway disorders被引用 22

一句话总结

本文揭示了标准异质性基准中的数据泄露和重复问题，提出多样化的新异质数据集，并发现标准GNN在大多数情况下优于异质性特定模型。

ABSTRACT

Node classification is a classical graph machine learning task on which Graph Neural Networks (GNNs) have recently achieved strong results. However, it is often believed that standard GNNs only work well for homophilous graphs, i.e., graphs where edges tend to connect nodes of the same class. Graphs without this property are called heterophilous, and it is typically assumed that specialized methods are required to achieve strong performance on such graphs. In this work, we challenge this assumption. First, we show that the standard datasets used for evaluating heterophily-specific models have serious drawbacks, making results obtained by using them unreliable. The most significant of these drawbacks is the presence of a large number of duplicate nodes in the datasets Squirrel and Chameleon, which leads to train-test data leakage. We show that removing duplicate nodes strongly affects GNN performance on these datasets. Then, we propose a set of heterophilous graphs of varying properties that we believe can serve as a better benchmark for evaluating the performance of GNNs under heterophily. We show that standard GNNs achieve strong results on these heterophilous graphs, almost always outperforming specialized models. Our datasets and the code for reproducing our experiments are available at https://github.com/yandex-research/heterophilous-graphs

研究动机与目标

• 揭示广泛使用的异质性基准的缺陷（例如重复节点和训练-测试泄漏）。
• 提出多样化、规模较大的异质性数据集，作为评估异质性下GNN的一种更好基准。
• 在新基准上对标准GNN和异质性特定模型进行经验比较，以评估进展。

提出的方法

• 批判性分析现有的异质性数据集并衡量其属性（调整后的同质性、标签信息量）。
• 鉴定导致训练-测试泄漏的重复项并量化移除后的性能变化。
• 创建五个跨领域的异质图，节点数在10K–50K之间，具有多样的结构属性。
• 在新数据集上对广泛的模型集合（标准GNN和异质性特定模型）进行基准测试。
• 研究自我节点与邻居嵌入分离作为影响异质性下性能的因素。

实验结果

研究问题

• RQ1 标准异质性基准是否因重复而产生数据泄漏，这如何影响模型排序？
• RQ2 在多样化的新异质基准上，标准GNN是否优于异质性特定模型？
• RQ3 在不同数据集上，分离自我和邻居嵌入是否始终能改善异质性下的性能？

主要发现

• squirrel 与 chameleon 数据集中的重复节点导致训练-测试泄漏；移除重复项会使模型性能发生巨大变化。
• 在新的异质性基准上，标准GNN在大多数数据集上超过异质性特定模型。
• 将自我与邻居嵌入分离的技巧（如 GAT-sep、GT-sep 等）在异质性下为标准GNN带来强劲效果。
• FSGNN 仍然是相对强劲的异质性特定模型，但并未持续占优；总体而言，许多异质性特定方法在所提基准上并不超过标准GNN。
• 基线中的最佳结果通常来自分离自我与邻居嵌入的模型，这表明这是异质性设置中的一项稳健策略。