[Paper Review] Attention Models in Graphs: A Survey

John Boaz Lee, Ryan A. Rossi+3 more|arXiv (Cornell University)|Jul 20, 2018

Advanced Graph Neural Networks58 citations

TL;DR

A comprehensive survey of graph attention models, introducing three taxonomies to organize methods by problem setting, attention type, and task, and outlining challenges and future directions.

ABSTRACT

Graph-structured data arise naturally in many different application domains. By representing data as graphs, we can capture entities (i.e., nodes) as well as their relationships (i.e., edges) with each other. Many useful insights can be derived from graph-structured data as demonstrated by an ever-growing body of work focused on graph mining. However, in the real-world, graphs can be both large - with many complex patterns - and noisy which can pose a problem for effective graph mining. An effective way to deal with this issue is to incorporate "attention" into graph mining solutions. An attention mechanism allows a method to focus on task-relevant parts of the graph, helping it to make better decisions. In this work, we conduct a comprehensive and focused survey of the literature on the emerging field of graph attention models. We introduce three intuitive taxonomies to group existing work. These are based on problem setting (type of input and output), the type of attention mechanism used, and the task (e.g., graph classification, link prediction, etc.). We motivate our taxonomies through detailed examples and use each to survey competing approaches from a unique standpoint. Finally, we highlight several challenges in the area and discuss promising directions for future work.

Motivation & Objective

• Provide a comprehensive overview of graph attention literature and motivate the need for attention in graphs.
• Introduce three intuitive taxonomies to categorize graph attention models.
• Survey existing methods from the perspectives of problem setting, attention mechanism, and task.
• Highlight challenges and propose promising directions for future work.

Proposed method

• Define formal graph types (homogeneous, heterogeneous, attributed, DAG) and a general graph attention framework.
• Introduce three taxonomies for organizing methods: problem setting, attention type, and application task.
• Survey methods by embedding type (node, edge, graph, hybrid) and by graph class, noting how attention is computed (similarity-based, attention-guided walks, etc.).
• Compare approaches and illustrate with examples and figures to motivate attention concepts in graphs.
• Discuss challenges and future research directions in graph attention.

Experimental results

Research questions

• RQ1 What are the main ways attention is applied to graphs across different problem settings?
• RQ2 How do different attention mechanisms compare in homogeneous vs heterogeneous graphs and across tasks like node/edge/graph embedding?
• RQ3 What are the key challenges and open directions in graph attention models?
• RQ4 How can taxonomy-driven analysis clarify the landscape of graph attention methods?

Key findings

• Attention helps improve signal-to-noise by focusing on task-relevant neighbors or substructures.
• Attention-based graph methods enable interpretability by highlighting influential parts of the graph.
• Three taxonomies effectively organize a wide range of graph attention models by problem setting, attention mechanism, and task.
• Several representative methods (e.g., GAT, AGNN, EAGCN) illustrate diverse attention formulations and their applicability to different graph types.
• There are still unsolved challenges in graph attention, with opportunities for future work in diverse graph classes and tasks.