[Paper Review] How Expressive are Transformers in Spectral Domain for Graphs?

Anson Bastos, Abhishek Nadgeri+4 more|arXiv (Cornell University)|Jan 23, 2022

Advanced Graph Neural Networks4 citations

TL;DR

This paper proposes FeTA, a spectral-domain attention mechanism that enables Transformers to attend over the full graph spectrum, overcoming the inherent limitation of vanilla Transformers in capturing high-frequency graph signals. Theoretical analysis shows spatial attention in Transformers is restricted to low-pass filtering, while FeTA achieves superior performance across graph classification, regression, and node classification benchmarks by learning task-specific spectral attention.

ABSTRACT

The recent works proposing transformer-based models for graphs have proven the inadequacy of Vanilla Transformer for graph representation learning. To understand this inadequacy, there is a need to investigate if spectral analysis of the transformer will reveal insights into its expressive power. Similar studies already established that spectral analysis of Graph neural networks (GNNs) provides extra perspectives on their expressiveness. In this work, we systematically study and establish the link between the spatial and spectral domain in the realm of the transformer. We further provide a theoretical analysis and prove that the spatial attention mechanism in the transformer cannot effectively capture the desired frequency response, thus, inherently limiting its expressiveness in spectral space. Therefore, we propose FeTA, a framework that aims to perform attention over the entire graph spectrum (i.e., actual frequency components of the graphs) analogous to the attention in spatial space. Empirical results suggest that FeTA provides homogeneous performance gain against vanilla transformer across all tasks on standard benchmarks and can easily be extended to GNN-based models with low-pass characteristics (e.g., GAT).

Motivation & Objective

• To theoretically analyze the expressive power of vanilla Transformers in the spectral domain for graph representation learning.
• To identify the fundamental limitation of spatial attention in Transformers—its inability to selectively attend to specific frequency components due to inherent low-pass filtering.
• To propose FeTA, a framework that performs attention over the entire graph spectrum, emulating spatial attention but in spectral space.
• To evaluate FeTA's generalization by integrating it into low-pass GNNs like GAT, demonstrating performance gains.
• To investigate the compatibility of FeTA with existing position encoding schemes, showing orthogonal improvements to spatial encoding methods.

Proposed method

• Formally establish an equivalence between spatial and spectral domains for Transformers using graph spectral theory.
• Theoretically prove that spatial attention in Transformers is constrained to low-pass frequency responses, limiting expressive power in spectral space.
• Design FeTA to perform self-attention over the graph Laplacian's eigenvectors (frequency components), enabling full-spectrum attention.
• Integrate FeTA into GNNs with low-pass filtering characteristics (e.g., GAT) to enhance their spectral modeling capability.
• Use learned attention weights over eigenmodes to dynamically filter signals across the entire frequency range, enabling task-specific spectral filtering.
• Evaluate FeTA with ablation studies and comparisons across multiple graph benchmarks, including ZINC, MolHIV, PATTERN, and CLUSTER.

Experimental results

Research questions

• RQ1 Is there a formal equivalence between the spatial and spectral domains for Transformers in graph representation learning?
• RQ2 What is the expressive power of Transformers in the spectral domain, and does it inherently restrict attention to low-frequency components?
• RQ3 Can a spectral-domain attention mechanism overcome the limitations of spatial attention in Transformers for graphs?
• RQ4 How does FeTA perform compared to vanilla Transformers and GNNs across diverse graph learning tasks?
• RQ5 To what extent do existing position encoding schemes interact with FeTA, and does FeTA provide complementary benefits?

Key findings

• Theoretical analysis confirms that the spatial attention mechanism in Transformers inherently restricts learning to low-pass frequency responses, limiting expressive power in spectral space.
• FeTA achieves consistent performance gains over vanilla Transformers across all evaluated graph classification, regression, and node classification benchmarks.
• When integrated into low-pass GNNs such as GAT, FeTA significantly improves performance by enabling full-spectrum attention, demonstrating generalizability.
• Attention heatmaps in spectral space reveal that FeTA learns to attend to both local clusters (low-pass behavior) and long-range interactions (high-pass behavior), depending on task needs.
• Position encoding schemes do not generalize across datasets, but FeTA consistently improves performance when replacing vanilla Transformers in state-of-the-art architectures, indicating orthogonal benefits.
• Interpretability analysis shows that FeTA’s spectral attention can be meaningfully interpreted as learning to aggregate nodes based on their spectral position, with low-frequency components emphasizing local homophily and high-frequency components enabling long-range interactions.