[Paper Review] Attention-based Graph Neural Network for Semi-supervised Learning

Kiran Koshy Thekumparampil, Chong Wang+2 more|arXiv (Cornell University)|Mar 10, 2018

Advanced Graph Neural NetworksComputer Science42 references250 citations

TL;DR

The paper introduces AGNN, an attention-based graph neural network that replaces traditional propagation with adaptive neighbor attention, achieving state-of-the-art results on citation networks while reducing model complexity.

ABSTRACT

Recently popularized graph neural networks achieve the state-of-the-art accuracy on a number of standard benchmark datasets for graph-based semi-supervised learning, improving significantly over existing approaches. These architectures alternate between a propagation layer that aggregates the hidden states of the local neighborhood and a fully-connected layer. Perhaps surprisingly, we show that a linear model, that removes all the intermediate fully-connected layers, is still able to achieve a performance comparable to the state-of-the-art models. This significantly reduces the number of parameters, which is critical for semi-supervised learning where number of labeled examples are small. This in turn allows a room for designing more innovative propagation layers. Based on this insight, we propose a novel graph neural network that removes all the intermediate fully-connected layers, and replaces the propagation layers with attention mechanisms that respect the structure of the graph. The attention mechanism allows us to learn a dynamic and adaptive local summary of the neighborhood to achieve more accurate predictions. In a number of experiments on benchmark citation networks datasets, we demonstrate that our approach outperforms competing methods. By examining the attention weights among neighbors, we show that our model provides some interesting insights on how neighbors influence each other.

Motivation & Objective

• Motivate semi-supervised learning on graphs with limited labels using graph structure and features.
• Show that a linear-propagation baseline (GLN) can match GCN performance, highlighting the propagation layer's importance.
• Propose AGNN with adaptive attention over neighbors to improve accuracy and interpretability.
• Demonstrate that AGNN achieves superior accuracy on standard citation network datasets compared to state-of-the-art methods.
• Provide insights into how learned attention weights reflect neighbor influence.

Proposed method

• Analyze Graph Neural Networks and identify that propagation layers largely drive performance while intermediate nonlinear layers contribute less.
• Define Graph Linear Network (GLN) to isolate propagation from nonlinearity and show it matches or approaches GCN performance.
• Introduce AGNN with a single scalar parameter per layer, using attention-weighted propagation: H^{(t+1)} = P^{(t)} H^{(t)}, where P^{(t)}_{ij} ∝ exp(β^{(t)} cos(H_i^{(t)}, H_j^{(t)})), and rows sum to one (softmax over neighbors).
• Compute node representations via an initial embedding XW^{(0)} with ReLU, followed by ell propagation layers, ending with a softmax classifier Z = softmax(H^{(ℓ+1)} W^{(1)}).
• Train all weights (W^{(0)}, W^{(1)}, β^{(t)}) with cross-entropy loss on labeled nodes.
• Provide complexity: O(ℓ d_h |E| + d_x d_h n).

Experimental results

Research questions

• RQ1 Can a simplified linear-propagation model achieve competitive performance with current GNNs on graph-based semi-supervised learning?
• RQ2 Do attention-based propagation layers improve accuracy by identifying and weighting more relevant neighbors in graph-structured data?
• RQ3 Does the proposed AGNN offer interpretability through learned attention weights that reveal neighbor influence patterns?
• RQ4 How does AGNN perform on standard citation-network benchmarks (CiteSeer, Cora, PubMed) compared with GCN and other baselines?

Key findings

• GLN (linear propagation) achieves accuracy comparable to or better than the best GCN on benchmark citation networks.
• AGNN achieves the best accuracy on CiteSeer, Cora, and PubMed in fixed-split experiments, with improvements exceeding the standard error.
• Across random splits and larger labeled-data regimes, AGNN consistently outperforms state-of-the-art baselines.
• Attention weights reveal that neighbors from the same class tend to receive higher attention, providing a degree of interpretability.
• Deeper propagation (ℓ up to 4) is feasible and beneficial for AGNN, thanks to reduced model complexity and absence of deep nonlinear layers.
• The attention mechanism focuses on neighboring nodes most relevant to the target node, improving classification, including for nodes misclassified by GCN.