[논문 리뷰] How Attentive are Graph Attention Networks?

Shaked Brody, Uri Alon외 1명|arXiv (Cornell University)|2021. 05. 30.

Advanced Graph Neural Networks참고 문헌 68인용 수 42

한 줄 요약

본 논문은 표준 Graph Attention Networks (GATs)가 동적 어텐션이 아닌 정적 어텐션을 구현함을 보이고, 이 한계를 이론적으로 증명하며, 동적 어텐션을 달성하고 다수의 벤치마크에서 우수한 성능을 발휘하는 간단한 연산 재배치인 GATv2를 도입한다.

ABSTRACT

Graph Attention Networks (GATs) are one of the most popular GNN architectures and are considered as the state-of-the-art architecture for representation learning with graphs. In GAT, every node attends to its neighbors given its own representation as the query. However, in this paper we show that GAT computes a very limited kind of attention: the ranking of the attention scores is unconditioned on the query node. We formally define this restricted kind of attention as static attention and distinguish it from a strictly more expressive dynamic attention. Because GATs use a static attention mechanism, there are simple graph problems that GAT cannot express: in a controlled problem, we show that static attention hinders GAT from even fitting the training data. To remove this limitation, we introduce a simple fix by modifying the order of operations and propose GATv2: a dynamic graph attention variant that is strictly more expressive than GAT. We perform an extensive evaluation and show that GATv2 outperforms GAT across 11 OGB and other benchmarks while we match their parametric costs. Our code is available at https://github.com/tech-srl/how_attentive_are_gats . GATv2 is available as part of the PyTorch Geometric library, the Deep Graph Library, and the TensorFlow GNN library.

연구 동기 및 목표

• Identify whether GATs compute dynamic attention as in classic attention mechanisms.
• Theoretically characterize the limitation of GAT’s attention as static and query-independent.
• Propose a minimal modification to transform GAT into a dynamic attention model (GATv2).
• Empirically compare GAT and GATv2 across diverse benchmarks to assess performance and robustness.
• Provide practical guidance on when to use GATv2 over GAT based on dataset properties.

제안 방법

• Define static vs. dynamic attention and formalize their properties.
• Prove that a standard GAT layer computes only static attention (Theorem 1).
• Propose GATv2 by rearranging the order of linear and nonlinearity operations to enable dynamic attention (Theorem 2).
• Evaluate on synthetic and real datasets (node-, link-, graph-prediction) including OGB benchmarks and QM9.
• Compare robustness to edge noise and performance with varying attention heads.
• Release implementation and integration with major frameworks.

실험 결과

연구 질문

• RQ1 Does standard GAT implement dynamic attention or only static attention?
• RQ2 Can a minimal modification to GAT enable dynamic attention without increasing parameter count dramatically?
• RQ3 Does GATv2 improve accuracy and robustness across diverse graph tasks compared to GAT?
• RQ4 In which benchmarks does dynamic attention yield the most benefit?

주요 결과

• GAT is limited to static attention where the ranking of neighbor importance is query-independent.
• GATv2, by reordering operations, achieves dynamic attention and strictly more expressiveness than GAT.
• GATv2 outperforms GAT across 12 benchmarks including Open Graph Benchmark tasks and QM9 properties.
• GATv2 shows greater robustness to structural noise than GAT.
• On the DictionaryLookup synthetic task, GAT fails to fit training data with a single head but GATv2 succeeds.
• In some tasks, non-attentive GNNs (GCN/GIN) can outperform attention-based models, highlighting task-dependent benefits of attention.