[Paper Review] Breaking the Limits of Message Passing Graph Neural Networks

Muhammet Balcılar, Pierre Héroux+4 more|arXiv (Cornell University)|Jun 8, 2021

Machine Learning and ELM28 citations

TL;DR

The paper designs spectral-domain graph convolutions with non-linear eigenvalue-based filters and receptive-field masking to surpass 1-WL expressiveness while keeping linear complexity, achieving 3-WL-equivalent power in practice (GNNML3) and enabling spectral flexibility.

ABSTRACT

Since the Message Passing (Graph) Neural Networks (MPNNs) have a linear complexity with respect to the number of nodes when applied to sparse graphs, they have been widely implemented and still raise a lot of interest even though their theoretical expressive power is limited to the first order Weisfeiler-Lehman test (1-WL). In this paper, we show that if the graph convolution supports are designed in spectral-domain by a non-linear custom function of eigenvalues and masked with an arbitrary large receptive field, the MPNN is theoretically more powerful than the 1-WL test and experimentally as powerful as a 3-WL existing models, while remaining spatially localized. Moreover, by designing custom filter functions, outputs can have various frequency components that allow the convolution process to learn different relationships between a given input graph signal and its associated properties. So far, the best 3-WL equivalent graph neural networks have a computational complexity in $\mathcal{O}(n^3)$ with memory usage in $\mathcal{O}(n^2)$, consider non-local update mechanism and do not provide the spectral richness of output profile. The proposed method overcomes all these aforementioned problems and reaches state-of-the-art results in many downstream tasks.

Motivation & Objective

• Motivate and quantify the limitations of standard MPNNs bound by 1-WL expressiveness.
• Propose spectral-domain graph convolution designs with non-linear eigenvalue functions to enhance expressive power.
• Maintain spatial locality and linear computational/memory complexity (except preprocessing).
• Introduce GNNML1 and GNNML3 to achieve 1-WL and beyond-1-WL expressive power respectively.
• Demonstrate empirical improvements on graph isomorphism-like tasks, substructure counting, and spectral filtering behavior.

Proposed method

• Formulate graph convolution as C^(s) = U diag(Phi_s(lambda)) U^T to design spectral supports with learned frequency responses.
• Introduce GNNML1 which preserves 1-WL power via a layer update combining node, neighbor aggregation, and feature-wise multiplication.
• Develop GNNML3 with spectral-domain supports expressed as power-series of the Laplacian/adjacency, enabling trace and element-wise multiplication effects (via masking and learned nonlinear frequency responses).
• Use pre-processing eigendecomposition to construct spectral supports, then propagate through multiple layers with learned MLPs (mlp_k) acting on edge-feature representations and node features.
• Provide algorithms for preprocessing (Algorithm 1) and forward computation (Algorithm 2) to realize the proposed models.
• Benchmark against 1-WL baselines (GCN, GAT, GraphSage, GIN, Chebnet) and a 3-WL baseline (PPGN) across graph isomorphism-like datasets and graphlet counting tasks.

Experimental results

Research questions

• RQ1 Can spectral-domain GNNs with non-linear eigenvalue functions surpass 1-WL expressiveness while maintaining linear time complexity?
• RQ2 Do designs like GNNML3 achieve practical 3-WL equivalence and enable counting of substructures beyond 1-WL-capable models?
• RQ3 Can the proposed models learn and utilize different spectral components (low/high/band-pass) for graph signals?
• RQ4 How do the models perform on tasks requiring higher-order graph distinctions and downstream graph classification/regression?
• RQ5 What are the limitations of these approaches in distinguishing strongly regular graphs and 3-WL-equivalent cases?

Key findings

• 1-WL-equivalent GNNs (e.g., GCN, GAT, GraphSage, GIN) remain limited, while Chebnet can be 1-WL-powerful under certain eigenvalue conditions.
• GNNML1 achieves 1-WL-level expressive power, matching 1-WL capabilities.
• GNNML3 breaks 1-WL limits and matches 3-WL power experimentally, enabling counting of triangle, 4-cycle, tailed triangle, and 3-star graphlets.
• Spectral designs with frequency-aware filters can realize trace and element-wise product-like effects within a linear-complexity framework (except preprocessing).
• Experiments show GNNML3 and PPGN distinguish graph pairs that are 1-WL equivalent but 3-WL distinguishable, and excel at graphlet counting; Chebnet distinguishes pairs with differing max eigenvalues in EXP datasets.
• Spectral-expressivity analysis indicates 1-WL-equivalent models largely learn low-pass filtering; GNNML1 can learn high-pass effects; GNNML3 achieves higher spectral flexibility (band-pass).
• The approach achieves state-of-the-art performance on several downstream graph problems while preserving locality and linear scalability.