[Paper Review] Graph Attention Networks

Petar Veličković, Guillem Cucurull+4 more|arXiv (Cornell University)|Oct 30, 2017

Advanced Graph Neural Networks8,277 citations

TL;DR

This paper introduces Graph Attention Networks (GATs), a novel graph neural network architecture that uses learnable self-attention mechanisms to dynamically assign different weights to neighboring nodes during message passing. By applying masked multi-head attention over node neighborhoods without requiring graph structure knowledge upfront, GATs achieve state-of-the-art performance on transductive and inductive node classification benchmarks, including Cora, Citeseer, PubMed, and a protein-protein interaction dataset with unseen graphs.

ABSTRACT

We present graph attention networks (GATs), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations. By stacking layers in which nodes are able to attend over their neighborhoods' features, we enable (implicitly) specifying different weights to different nodes in a neighborhood, without requiring any kind of costly matrix operation (such as inversion) or depending on knowing the graph structure upfront. In this way, we address several key challenges of spectral-based graph neural networks simultaneously, and make our model readily applicable to inductive as well as transductive problems. Our GAT models have achieved or matched state-of-the-art results across four established transductive and inductive graph benchmarks: the Cora, Citeseer and Pubmed citation network datasets, as well as a protein-protein interaction dataset (wherein test graphs remain unseen during training).

Motivation & Objective

• To address the limitations of spectral-based graph neural networks that depend on fixed graph structures and costly matrix operations.
• To develop a fully differentiable, efficient, and inductive message-passing mechanism for graph-structured data.
• To enable models to assign varying importance to different neighbors in a neighborhood without prior knowledge of the graph structure.
• To improve performance on both transductive and inductive node classification tasks, especially on unseen graphs.
• To explore the benefits of attention mechanisms in capturing structural dependencies in irregular graph domains.

Proposed method

• Proposes a graph attention layer that computes attention coefficients between a node and its neighbors using a shared, learnable parameter matrix.
• Applies masked multi-head attention to stabilize training and allow the model to attend to different subspaces of the feature representation.
• Uses a softmax-normalized attention mechanism to compute weighted sums of neighbor features, enabling dynamic, adaptive aggregation.
• Stacks multiple graph attention layers to learn deep hierarchical representations, with residual connections for training stability.
• Employs a parameter-sharing mechanism across edges, making the model scalable and parallelizable.
• Introduces a constant-attention variant (Const-GAT) to ablate the contribution of learnable attention weights.

Experimental results

Research questions

• RQ1 Can a self-attention mechanism be effectively adapted to graph-structured data to improve node representation learning?
• RQ2 Does learning dynamic attention weights over neighbors lead to better performance than fixed aggregation schemes like GCN?
• RQ3 Can the model generalize to inductive tasks, including graphs not seen during training?
• RQ4 How does the attention mechanism improve interpretability and feature learning in graph neural networks?
• RQ5 What is the impact of multi-head attention on model performance and robustness?

Key findings

• GAT achieves state-of-the-art performance on the Cora citation network, attaining an F1 score of 83.0 ± 0.7%, outperforming GCN by 1.5%.
• On Citeseer, GAT achieves 72.5 ± 0.7% F1, a 1.6% improvement over GCN, demonstrating the benefit of adaptive neighbor weighting.
• On PubMed, GAT matches GCN’s performance at 79.0 ± 0.3%, confirming strong results on large-scale citation networks.
• On the inductive PPI dataset, GAT achieves 97.3 ± 0.002% accuracy, a 20.5% improvement over the best GraphSAGE result and a 3.9% gain over the constant-attention baseline (Const-GAT).
• Visualization of feature representations using t-SNE shows clear class-wise clustering, indicating strong discriminative power.
• Ablation studies confirm that the attention mechanism is crucial, as the constant-attention variant (Const-GAT) significantly underperforms the full GAT model.