[Paper Review] Fine-grained Expressivity of Graph Neural Networks

Jan Böker, Ron Levie+3 more|arXiv (Cornell University)|Jun 6, 2023

Advanced Memory and Neural Computing4 citations

TL;DR

This paper introduces a continuous, topological framework for measuring the expressive power of message-passing graph neural networks (MPNNs) using graphons, extending the 1-dimensional Weisfeiler–Leman (1-WL) test to a fine-grained, metric-based characterization. It establishes equivalence between MPNN output similarity, tree distance, and homomorphism densities, proving that MPNNs' ability to distinguish graphs is fully captured by these continuous metrics, with untrained MPNNs achieving competitive performance on graph classification tasks.

ABSTRACT

Numerous recent works have analyzed the expressive power of message-passing graph neural networks (MPNNs), primarily utilizing combinatorial techniques such as the $1$-dimensional Weisfeiler-Leman test ($1$-WL) for the graph isomorphism problem. However, the graph isomorphism objective is inherently binary, not giving insights into the degree of similarity between two given graphs. This work resolves this issue by considering continuous extensions of both $1$-WL and MPNNs to graphons. Concretely, we show that the continuous variant of $1$-WL delivers an accurate topological characterization of the expressive power of MPNNs on graphons, revealing which graphs these networks can distinguish and the level of difficulty in separating them. We identify the finest topology where MPNNs separate points and prove a universal approximation theorem. Consequently, we provide a theoretical framework for graph and graphon similarity combining various topological variants of classical characterizations of the $1$-WL. In particular, we characterize the expressive power of MPNNs in terms of the tree distance, which is a graph distance based on the concept of fractional isomorphisms, and substructure counts via tree homomorphisms, showing that these concepts have the same expressive power as the $1$-WL and MPNNs on graphons. Empirically, we validate our theoretical findings by showing that randomly initialized MPNNs, without training, exhibit competitive performance compared to their trained counterparts. Moreover, we evaluate different MPNN architectures based on their ability to preserve graph distances, highlighting the significance of our continuous $1$-WL test in understanding MPNNs' expressivity.

Motivation & Objective

• To address the limitation of binary graph isomorphism in measuring MPNN expressivity by introducing a continuous, metric-based characterization of graph similarity.
• To formalize the expressive power of MPNNs on graphons using topological variants of the 1-WL test.
• To establish equivalence between MPNN output similarity, tree distance, and substructure counts via tree homomorphisms.
• To provide a universal approximation theorem for MPNNs in the context of graphons.
• To validate theoretically derived metrics empirically by showing untrained MPNNs achieve competitive performance on graph-level prediction tasks.

Proposed method

• Extends the 1-WL test to graphons using iterated degree measures and compact operators.
• Introduces two continuous metrics, δ_P (Prokhorov) and δ_W (unbalanced Wasserstein), to metrize the topology of iterated degree measures.
• Proves that two graphons are close in δ_P or δ_W if and only if all MPNNs with bounded Lipschitz constant and layers produce close outputs.
• Links the metrics to tree distance δ_T□ and tree homomorphism densities, showing they are equivalent in expressive power.
• Uses the theory of fractional isomorphisms and graphon limits to derive a universal approximation result for MPNNs.
• Empirically evaluates untrained MPNNs on standard graph benchmarks and compares their performance to trained models.

Experimental results

Research questions

• RQ1 How can the expressive power of MPNNs be characterized in a continuous, fine-grained manner beyond binary isomorphism?
• RQ2 What is the finest topology in which MPNNs separate graphons, and how does this relate to classical 1-WL and graphon structure?
• RQ3 Are metrics based on MPNN outputs, tree distances, and tree homomorphism densities equivalent in capturing MPNN expressivity on graphons?
• RQ4 Can untrained MPNNs achieve competitive performance on graph classification, and what does this imply about their inherent representational capacity?
• RQ5 How do the proposed continuous metrics δ_P and δ_W compare to existing graph distances like the WL distance in practical classification tasks?

Key findings

• The proposed metrics δ_P and δ_W are equivalent to the tree distance δ_T□ and to MPNN output similarity for all MPNNs with bounded Lipschitz constant and number of layers.
• Untrained MPNNs achieve competitive performance on graph-level prediction tasks (e.g., 81.78% ± 3.29 on Mutag), suggesting their initial weights already encode meaningful structural information.
• The Euclidean distance between MPNN embeddings serves as an efficient lower bound for the graph distances δ_P and δ_W, with significantly lower computational cost than direct distance computation.
• The 1-NN classifier using δ_W achieves 87.89% ± 4.11 accuracy on Mutag and 66.50% ± 4.15 on Imdb-Binary, comparable to the WL distance (91.1% ± 4.3 and 69.4% ± 3.9).
• Increasing the number of layers initially improves untrained MPNN performance but eventually degrades it, indicating an optimal depth for structural representation.
• The theoretical framework provides a universal approximation theorem for MPNNs on graphons, showing that their expressive power is fully characterized by the continuous 1-WL test.