[Paper Review] Spectral Graph Attention Network with Fast Eigen-approximation

Heng Chang, Yu Rong+5 more|arXiv (Cornell University)|Mar 16, 2020

Advanced Graph Neural Networks50 references4 citations

TL;DR

This paper proposes Spectral Graph Attention Network (SpGAT), a novel graph neural network that applies attention mechanisms to spectral graph wavelets, enabling efficient modeling of both low- and high-frequency components for improved global and local representation learning. It further introduces SpGAT-Cheby, a fast approximation variant using Chebyshev polynomials that reduces computational cost by up to 7.9× on large graphs while achieving state-of-the-art performance on citation benchmarks.

ABSTRACT

Variants of Graph Neural Networks (GNNs) for representation learning have been proposed recently and achieved fruitful results in various fields. Among them, Graph Attention Network (GAT) first employs a self-attention strategy to learn attention weights for each edge in the spatial domain. However, learning the attentions over edges can only focus on the local information of graphs and greatly increases the computational costs. In this paper, we first introduce the attention mechanism in the spectral domain of graphs and present Spectral Graph Attention Network (SpGAT) that learns representations for different frequency components regarding weighted filters and graph wavelets bases. In this way, SpGAT can better capture global patterns of graphs in an efficient manner with much fewer learned parameters than that of GAT. Further, to reduce the computational cost of SpGAT brought by the eigen-decomposition, we propose a fast approximation variant SpGAT-Cheby. We thoroughly evaluate the performance of SpGAT and SpGAT-Cheby in semi-supervised node classification tasks and verify the effectiveness of the learned attentions in the spectral domain.

Motivation & Objective

• To address the limitation of spatial-domain attention in GNNs, which focuses only on local neighborhood information and incurs high computational cost.
• To enable global structural modeling in graphs by extending the attention mechanism to the spectral domain using graph wavelets and frequency components.
• To reduce the computational burden of spectral decomposition in spectral GNNs through a fast Chebyshev polynomial approximation.
• To demonstrate that learning attention over spectral components leads to better performance and parameter efficiency than spatial attention mechanisms.

Proposed method

• SpGAT decomposes graph signals into low- and high-frequency components using graph wavelets derived from the Laplacian's eigendecomposition.
• It applies separate learnable attention weights to low- and high-frequency components, allowing distinct modeling of local and global graph structures.
• The model uses a pooling function and activation to aggregate and transform the attended spectral features into final node representations.
• SpGAT-Cheby replaces exact eigen-decomposition with a Chebyshev polynomial approximation to accelerate spectral wavelet computation.
• The approximation leverages the fact that Chebyshev polynomials can efficiently approximate spectral functions over the eigenvalue interval.
• The method is trained end-to-end in a semi-supervised node classification setting using cross-entropy loss.

Experimental results

Research questions

• RQ1 Can attention mechanisms applied in the spectral domain improve global representation learning in graphs compared to spatial-domain attention?
• RQ2 Does separating low- and high-frequency components in the spectral domain lead to better performance and parameter efficiency?
• RQ3 Can Chebyshev polynomial approximation significantly reduce the computational cost of spectral wavelet computation without sacrificing performance?
• RQ4 What is the optimal proportion of low-frequency components for effective graph representation learning?

Key findings

• SpGAT achieved state-of-the-art performance on all five citation datasets, outperforming 14 competitive baselines, including GAT and GCN.
• On the PubMed dataset, SpGAT-Cheby achieved 92.1% accuracy with a 7.9× speedup in spectral wavelet computation compared to exact eigen-decomposition.
• The optimal proportion of low-frequency components was found to be only 5–15% across datasets, indicating that a small number of components suffice for effective signal reconstruction.
• SpGAT-Cheby reduced training time by nearly half compared to GAT and matched the efficiency of VanillaGCN, demonstrating strong scalability.
• Ablation studies confirmed that attention over spectral components significantly improves performance, with the best results achieved when low-frequency components were treated with a small, fixed proportion.
• The model's performance was robust across different datasets, with SpGAT-Cheby achieving the highest accuracy on PubMed (92.8%) and Cora (83.7%).