[Paper Review] A Theoretical Comparison of Graph Neural Network Extensions

Pál András Papp, Roger Wattenhofer|arXiv (Cornell University)|Jan 30, 2022

Advanced Graph Neural Networks4 citations

TL;DR

This paper introduces and compares three novel GNN extensions—subgraph preprocessing (Sₖ), neighborhood preprocessing (Nₖ), and node marking (Mₖ)—to overcome the limitations of standard GNNs bounded by the 1-WL test. It establishes a new hierarchy of expressiveness, showing that Sₖ is strictly more expressive than Mₖ for k ≥ 3, and that Mₖ can distinguish graphs that Nₖ cannot, while also proving that Sₖ cannot count (k+1)-cycles, highlighting fundamental limits of subgraph-based approaches.

ABSTRACT

We study and compare different Graph Neural Network extensions that increase the expressive power of GNNs beyond the Weisfeiler-Leman test. We focus on (i) GNNs based on higher order WL methods, (ii) GNNs that preprocess small substructures in the graph, (iii) GNNs that preprocess the graph up to a small radius, and (iv) GNNs that slightly perturb the graph to compute an embedding. We begin by presenting a simple improvement for this last extension that strictly increases the expressive power of this GNN variant. Then, as our main result, we compare the expressiveness of these extensions to each other through a series of example constructions that can be distinguished by one of the extensions, but not by another one. We also show negative examples that are particularly challenging for each of the extensions, and we prove several claims about the ability of these extensions to count cliques and cycles in the graph.

Motivation & Objective

• To identify and formalize three main approaches in the literature for enhancing GNN expressiveness: subgraph preprocessing (Sₖ), neighborhood preprocessing (Nₖ), and node marking (Mₖ).
• To develop a new, more nuanced hierarchy of expressive power for GNN extensions, moving beyond the coarse k-WL hierarchy.
• To compare the relative expressive power of these extensions using explicit graph constructions that can be distinguished by one but not another.
• To analyze the ability of each extension to count specific substructures, such as cliques and cycles, under both graph and node classification settings.

Proposed method

• The authors define three GNN extensions: Sₖ processes all subgraphs of size up to k, Nₖ processes k-hop induced neighborhoods, and Mₖ marks or removes k nodes for symmetry breaking.
• They construct specific graph pairs that are indistinguishable to one extension but distinguishable to another, using cycles of varying lengths and a central node to break symmetry.
• For each extension, they prove invariance under certain graph transformations (e.g., cycle length changes) to show limitations in distinguishing graphs.
• They use a combination of theoretical proofs and lemmas (e.g., Lemma C.1, C.2) to demonstrate that certain embeddings remain identical across graphs under specific conditions, such as when subgraphs or neighborhoods are isomorphic.
• They compare the extensions not only to each other but also to the classical k-WL hierarchy, showing that k-WL is too coarse for practical comparison.
• They analyze subgraph counting capabilities by constructing graphs with multiple disjoint cycles of different lengths, proving that Sₖ cannot count (k+1)-cycles and Nₖ cannot count (2k+2)-cycles.

Experimental results

Research questions

• RQ1 How do the expressive powers of Sₖ, Nₖ, and Mₖ compare to each other, and is there a strict ordering or mutual incomparability?
• RQ2 Can the k-WL hierarchy serve as a meaningful benchmark for evaluating GNN extensions, or is it too coarse for practical comparisons?
• RQ3 To what extent can Sₖ, Nₖ, and Mₖ count specific substructures like cycles or cliques in a graph?
• RQ4 Is there a GNN extension that is strictly more expressive than others in a given class, such as Mₖ being more expressive than previous symmetry-breaking methods?
• RQ5 Can we construct graph pairs that are distinguishable by one extension but not by another, and what does this reveal about the limitations of each approach?

Key findings

• Sₖ is strictly more expressive than Mₖ for k ≥ 3, as demonstrated by a construction where Sₖ can distinguish two graphs with cycles of length 2k+1, while Mₖ cannot.
• Mₖ is strictly more expressive than Nₖ, as shown by a construction where Mₖ can distinguish two graphs with long cycles, while Nₖ cannot.
• Sₖ cannot count (k+1)-cycles, even when using subgraph counts, due to isomorphic subgraphs in graphs with cycles of different lengths.
• Nₖ cannot count (2k+2)-cycles, as the k-hop neighborhoods of nodes in such graphs are isomorphic across different cycle lengths.
• For k ≥ 3, Sₖ can distinguish graphs that Mₖ cannot, but the reverse is not true, establishing a strict superiority of Sₖ over Mₖ in this regime.
• The paper shows that Sₖ is more expressive than Nₖ in the graph classification setting, with Sₖ₊₃ > Nₖ, though the result is weaker than for other comparisons.