[Paper Review] Direct Multi-hop Attention based Graph Neural Network

Guangtao Wang, Rex Ying+2 more|arXiv (Cornell University)|Sep 29, 2020

Advanced Graph Neural Networks22 citations

TL;DR

DAGN proposes a novel Graph Neural Network that extends self-attention to capture multi-hop neighborhood context by diffusing attention scores across paths, enabling long-range dependencies in every layer. It achieves state-of-the-art performance with up to 5.7% relative error reduction on citation networks and SOTA results on knowledge graph completion benchmarks.

ABSTRACT

Introducing self-attention mechanism in graph neural networks (GNNs) achieved state-of-the-art performance for graph representation learning. However, at every layer, attention is only computed between two connected nodes and depends solely on the representation of both nodes. This attention computation cannot account for the multi-hop neighbors which supply graph structure context information and have influence on the node representation learning as well. In this paper, we propose Direct Multi-hop Attention based Graph neural Network (DAGN) for graph representation learning, a principled way to incorporate multi-hop neighboring context into attention computation, enabling long-range interactions at every layer. To compute attention between nodes that are multiple hops away, DAGN diffuses the attention scores from neighboring nodes to non-neighboring nodes, thus increasing the receptive field for every message passing layer. Unlike previous methods, DAGN uses a diffusion prior on attention values, to efficiently account for all paths between the pair of nodes when computing multi-hop attention weights. This helps DAGN capture large-scale structural information in a single layer, and learn more informative attention distribution. Experimental results on standard semi-supervised node classification as well as the knowledge graph completion show that DAGN achieves state-of-the-art results: DAGN achieves up to 5.7% relative error reduction over the previous state-of-the-art on Cora, Citeseer, and Pubmed. DAGN also obtains the best performance on a large-scale Open Graph Benchmark dataset. On knowledge graph completion DAGN advances state-of-the-art on WN18RR and FB15k-237 across four different performance metrics.

Motivation & Objective

• To address the limitation of standard GNNs that restrict attention to only immediate neighbors, missing structural context from multi-hop neighbors.
• To enable long-range message passing in GNNs by incorporating structural information from all paths between nodes.
• To develop a differentiable, efficient mechanism to compute attention over multi-hop paths without explicit path enumeration.
• To improve node representation learning by leveraging a diffusion prior on attention values that aggregates influence across all paths.
• To achieve state-of-the-art performance on semi-supervised node classification and knowledge graph completion tasks.

Proposed method

• Introduces a diffusion mechanism that propagates attention scores from neighboring nodes to non-adjacent nodes, expanding the receptive field beyond direct edges.
• Employs a diffusion prior on attention weights that aggregates influence across all possible paths between node pairs, modeling multi-hop dependencies.
• Computes multi-hop attention by diffusing attention scores through the graph using a learnable diffusion matrix, avoiding explicit path enumeration.
• Integrates the diffused attention into the message-passing framework of GNNs, allowing each layer to attend to distant nodes with structural context.
• Uses a differentiable attention mechanism that combines node representations and diffused attention to produce enhanced node embeddings.
• Applies the method in a multi-layer GNN architecture, where each layer benefits from enriched, long-range attention distributions.

Experimental results

Research questions

• RQ1 Can multi-hop structural context be effectively incorporated into graph attention networks without increasing computational complexity?
• RQ2 How does modeling attention over all paths between nodes improve node representation learning in GNNs?
• RQ3 To what extent does the proposed diffusion-based attention mechanism outperform standard GNNs and existing multi-hop methods on node classification and knowledge graph completion?
• RQ4 Can the method generalize across diverse graph types, including citation networks and large-scale knowledge graphs?
• RQ5 Does the attention diffusion mechanism lead to more robust and informative node embeddings compared to baseline GNNs?

Key findings

• DAGN achieves up to 5.7% relative error reduction over the previous state-of-the-art on the Cora, Citeseer, and PubMed citation networks.
• On the large-scale Open Graph Benchmark, DAGN achieves the best performance among all compared methods.
• On knowledge graph completion, DAGN sets new state-of-the-art results on WN18RR and FB15k-237 across four evaluation metrics.
• The model demonstrates significant improvements in node classification accuracy by capturing long-range dependencies through diffused attention.
• The diffusion-based attention mechanism enables effective modeling of structural context without explicit path enumeration, maintaining efficiency.
• The method generalizes well across diverse graph types, including transductive and inductive learning settings.