[Paper Review] Graph External Attention Enhanced Transformer

Jianqing Liang, Min Chen+1 more|arXiv (Cornell University)|May 31, 2024

EEG and Brain-Computer Interfaces4 citations

TL;DR

This paper proposes Graph External Attention Enhanced Transformer (GEAET), a novel Transformer-based architecture that enhances graph representation learning by integrating inter-graph correlations through a lightweight Graph External Attention (GEA) mechanism. By leveraging external node/edge key-value units, GEAET captures global structural similarities across graphs, achieving state-of-the-art performance on benchmark datasets while reducing reliance on positional encoding.

ABSTRACT

The Transformer architecture has recently gained considerable attention in the field of graph representation learning, as it naturally overcomes several limitations of Graph Neural Networks (GNNs) with customized attention mechanisms or positional and structural encodings. Despite making some progress, existing works tend to overlook external information of graphs, specifically the correlation between graphs. Intuitively, graphs with similar structures should have similar representations. Therefore, we propose Graph External Attention (GEA) -- a novel attention mechanism that leverages multiple external node/edge key-value units to capture inter-graph correlations implicitly. On this basis, we design an effective architecture called Graph External Attention Enhanced Transformer (GEAET), which integrates local structure and global interaction information for more comprehensive graph representations. Extensive experiments on benchmark datasets demonstrate that GEAET achieves state-of-the-art empirical performance. The source code is available for reproducibility at: https://github.com/icm1018/GEAET.

Motivation & Objective

• To address the limitation of existing graph representation models that focus only on internal graph structure and ignore inter-graph correlations.
• To improve long-range dependency modeling and mitigate over-squashing and over-smoothing issues in GNNs.
• To design a self-attention mechanism that implicitly learns structural similarities across graphs without relying heavily on positional encodings.
• To integrate local structural information with global inter-graph interactions for more comprehensive graph representations.
• To achieve state-of-the-art performance on standard graph benchmark datasets while maintaining interpretability and efficiency.

Proposed method

• Proposes Graph External Attention (GEA), a novel attention mechanism that uses external key-value units from other graphs to model inter-graph correlations.
• Introduces a multi-head attention mechanism where query vectors are derived from a single graph, while key-value pairs are drawn from a set of external graphs.
• Designs GEAET as a hybrid architecture combining a GCN or GIN encoder for local structure learning with the GEA module for global interaction modeling.
• Employs learnable external key-value units for nodes and edges, enabling the model to implicitly capture structural similarities across graphs.
• Reduces dependency on positional encoding by leveraging structural similarity through external attention, improving generalization.
• Uses a parallel architecture with GCN and attention blocks, allowing joint optimization of local and global representations.

Experimental results

Research questions

• RQ1 Can modeling inter-graph structural correlations improve graph representation learning beyond intra-graph message passing?
• RQ2 How does Graph External Attention compare to standard self-attention in terms of performance and dependence on positional encoding?
• RQ3 To what extent can GEAET mitigate over-squashing in graph neural networks on long-range dependency tasks?
• RQ4 What is the contribution of external node, edge, and shared key-value units to the overall model performance?
• RQ5 Does the integration of global inter-graph attention lead to state-of-the-art performance on standard graph benchmarks?

Key findings

• GEAET achieves state-of-the-art performance on multiple benchmark datasets, including ZINC, Peptides-Struct, and TreeNeighbourMatch.
• The model outperforms standard Transformers and GNNs on long-range dependency tasks, generalizing effectively up to a tree depth of r=7, whereas GNNs fail at r=4.
• GEAET reduces reliance on positional encoding: it achieves strong performance without any positional encoding, while Transformers degrade significantly without it.
• Ablation studies confirm that all components—external node, edge, and shared units—contribute to performance, with the full configuration achieving 98.513% accuracy on MNIST and 86.993% on PATTERN.
• The use of 8 attention heads in GEANet yields the best performance, contrasting with Transformers, which perform best with a single head, indicating different inductive biases in attention mechanisms.
• Laplacian Positional Encoding slightly improves GEANet performance, while Random Walk Positional Encoding degrades it, further highlighting GEANET's reduced need for such encodings.