[Paper Review] GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs

Jiani Zhang, Xingjian Shi+4 more|arXiv (Cornell University)|Mar 20, 2018

Advanced Graph Neural Networks19 references314 citations

TL;DR

GaAN introduces a gated multi-head attention graph aggregator where a small convolutional sub-network gates each attention head, improving inductive node classification and enabling a Graph Gated Recurrent Unit (GGRU) for spatiotemporal forecasting.

ABSTRACT

We propose a new network architecture, Gated Attention Networks (GaAN), for learning on graphs. Unlike the traditional multi-head attention mechanism, which equally consumes all attention heads, GaAN uses a convolutional sub-network to control each attention head's importance. We demonstrate the effectiveness of GaAN on the inductive node classification problem. Moreover, with GaAN as a building block, we construct the Graph Gated Recurrent Unit (GGRU) to address the traffic speed forecasting problem. Extensive experiments on three real-world datasets show that our GaAN framework achieves state-of-the-art results on both tasks.

Motivation & Objective

• Motivate and design a graph aggregator that learns which attention heads are most informative for a center node’s neighborhood.
• Develop GaAN, a gated multi-head attention aggregator, to improve node representation learning on large graphs.
• Extend GaAN to recurrent structures (GGRU) for spatiotemporal forecasting on graphs.
• Improve efficiency for training on large graphs via improved sampling and neighbor merging strategies.
• Demonstrate state-of-the-art performance on inductive node classification and traffic speed forecasting datasets.

Proposed method

• Propose GaAN: a multi-head attention aggregator with a gating sub-network that assigns a gate scalar to each head per center node.
• Compute gates g_i^(k) using a lightweight convolutional network that aggregates center-node and neighbor features (Eq. 3).
• Integrate gates into the standard multi-head attention aggregation (Eq. 2) to modulate each head’s contribution.
• Offer a unified framework to convert any graph aggregator into a recurrent unit (GGRU) for spatiotemporal forecasting (Graph GRU equations in Section 6.1).
• Propose memory-efficient neighbor sampling and merging (similar to GraphSAGE) to handle large graphs during training.
• Compare GaAN to multiple baselines (attention-based, pooling-based, and pairwise-sum aggregators) on PPI and Reddit for node classification, and METR-LA for traffic forecasting.

Experimental results

Research questions

• RQ1 Does adding gates to multi-head attention improve performance over standard multi-head attention on inductive node classification tasks?
• RQ2 Can GaAN outperform pooling and pairwise-sum graph aggregators on real-world large graphs?
• RQ3 Does GaAN generalize to a Graph GRU framework for spatiotemporal forecasting?
• RQ4 What is the impact of gate design, number of attention heads, and sampling strategy on performance and efficiency?

Key findings

• GaAN achieves state-of-the-art micro F1 on PPI and Reddit inductive node classification benchmarks compared to baseline aggregators and attention-only models.
• Gate-enhanced attention (GaAN) consistently outperforms the standard multi-head attention with the same number of heads, showing the benefit of head-wise gating.
• Larger output dimensions and more attention heads can improve performance, with dataset-dependent trends observed (PPI vs Reddit).
• A gating network with small dimensionality (d_m = 64) suffices to provide meaningful head-wise gates with low overhead.
• GaAN-based GGRU improves spatiotemporal forecasting capability, enabling a unified approach to graph-based sequence prediction.
• Experimental results indicate strong performance gains over baselines across datasets and tasks.