[Paper Review] Graph Kernel Attention Transformers

Krzysztof Choromański, Han Lin+2 more|arXiv (Cornell University)|Jul 16, 2021

Advanced Graph Neural Networks42 references4 citations

TL;DR

This paper proposes Graph Kernel Attention Transformers (GKATs), a novel GNN architecture that integrates graph kernels, attention mechanisms with structural priors, and low-rank implicit attention via random walk-based kernel decomposition. GKATs achieve linear computational scaling with graph size, model long-range dependencies efficiently in shallow architectures, and outperform SOTA GNNs across diverse benchmarks including motif detection, social networks, and bioinformatics tasks.

ABSTRACT

We introduce a new class of graph neural networks (GNNs), by combining several concepts that were so far studied independently - graph kernels, attention-based networks with structural priors and more recently, efficient Transformers architectures applying small memory footprint implicit attention methods via low rank decomposition techniques. The goal of the paper is twofold. Proposed by us Graph Kernel Attention Transformers (or GKATs) are much more expressive than SOTA GNNs as capable of modeling longer-range dependencies within a single layer. Consequently, they can use more shallow architecture design. Furthermore, GKAT attention layers scale linearly rather than quadratically in the number of nodes of the input graphs, even when those graphs are dense, requiring less compute than their regular graph attention counterparts. They achieve it by applying new classes of graph kernels admitting random feature map decomposition via random walks on graphs. As a byproduct of the introduced techniques, we obtain a new class of learnable graph sketches, called graphots, compactly encoding topological graph properties as well as nodes' features. We conducted exhaustive empirical comparison of our method with nine different GNN classes on tasks ranging from motif detection through social network classification to bioinformatics challenges, showing consistent gains coming from GKATs.

Motivation & Objective

• To address the limitations of existing GNNs in modeling long-range dependencies efficiently.
• To reduce the quadratic computational cost of standard graph attention mechanisms in dense graphs.
• To develop a learnable, compact graph sketching method that encodes topological and feature information.
• To unify graph kernels, attention mechanisms, and efficient Transformer architectures into a single expressive framework.
• To achieve state-of-the-art performance with shallow architectures through improved inductive bias and expressivity.

Proposed method

• GKATs employ a new class of graph kernels derived from random walks on graphs, enabling efficient random feature map decomposition.
• These kernels are integrated into an attention mechanism, forming a structured inductive bias that enhances modeling of structural dependencies.
• The attention mechanism uses low-rank decomposition to achieve linear complexity in the number of nodes, even for dense graphs.
• A new learnable graph sketching technique, called 'graphots', is introduced to compactly encode topological and feature properties of graphs.
• The architecture combines the inductive bias of graph kernels with the representational power of Transformers, enabling shallow yet highly expressive models.
• The method supports end-to-end training with differentiable attention weights and kernel parameters, allowing joint optimization of structural and feature representations.

Experimental results

Research questions

• RQ1 Can graph kernel-based attention mechanisms achieve linear complexity in graph size while maintaining or improving expressivity?
• RQ2 To what extent can GKATs model long-range dependencies in a single layer compared to standard GNNs?
• RQ3 How does the integration of random walk-based graph kernels with attention improve generalization across diverse graph learning tasks?
• RQ4 Can the proposed graphots sketching method effectively capture topological and feature information in a compact, learnable form?
• RQ5 Does the GKAT architecture consistently outperform SOTA GNNs across diverse benchmarks, including bioinformatics and social network tasks?

Key findings

• GKATs achieve linear computational complexity with respect to the number of nodes, even in dense graphs, unlike standard GNNs with quadratic complexity.
• The model demonstrates superior performance in modeling long-range dependencies, enabling effective representation learning with shallow architectures.
• GKATs outperform nine different GNN classes across diverse tasks, including motif detection, social network classification, and bioinformatics benchmarks.
• The introduced graphots provide a compact, learnable sketch of graph structure and node features, enhancing representation efficiency.
• Empirical evaluations confirm consistent gains across all evaluated tasks, validating the effectiveness of the proposed architectural integration.
• The method maintains high expressivity while reducing memory and compute demands, making it suitable for large-scale graph learning.