[Paper Review] The Descriptive Complexity of Graph Neural Networks

Martin Grohe|arXiv (Cornell University)|Mar 8, 2023

Machine Learning and Algorithms4 citations

TL;DR

This paper establishes a precise logical and circuit-theoretic characterization of Graph Neural Networks (GNNs), showing that bounded-depth GNNs with real weights and standard activation functions compute exactly the queries definable in the guarded fragment of first-order logic with counting (GFO+C). With random initialization and global readout, they capture the complexity class TC⁰, revealing a tight correspondence between GNNs and classical descriptive complexity theory despite their continuous, real-valued computations.

ABSTRACT

We analyse the power of graph neural networks (GNNs) in terms of Boolean circuit complexity and descriptive complexity. We prove that the graph queries that can be computed by a polynomial-size bounded-depth family of GNNs are exactly those definable in the guarded fragment GFO+C of first-order logic with counting and with built-in relations. This puts GNNs in the circuit complexity class (non-uniform) $ ext{TC}^0$. Remarkably, the GNN families may use arbitrary real weights and a wide class of activation functions that includes the standard ReLU, logistic "sigmoid", and hyperbolic tangent functions. If the GNNs are allowed to use random initialisation and global readout (both standard features of GNNs widely used in practice), they can compute exactly the same queries as bounded depth Boolean circuits with threshold gates, that is, exactly the queries in $ ext{TC}^0$. Moreover, we show that queries computable by a single GNN with piecewise linear activations and rational weights are definable in GFO+C without built-in relations. Therefore, they are contained in uniform $ ext{TC}^0$.

Motivation & Objective

• To characterize the logical and computational expressiveness of Graph Neural Networks (GNNs) in terms of descriptive complexity theory.
• To determine the exact class of graph queries computable by polynomial-size, bounded-depth GNNs with real weights and standard activation functions.
• To investigate the impact of random initialization and global readout on GNN expressiveness, particularly in relation to Boolean circuit classes.
• To establish a formal bridge between continuous, real-number-based GNNs and discrete, finite models of computation such as first-order logic and TC⁰ circuits.
• To explore the role of aggregation functions, activation functions, and weight types (rational vs. real) in determining GNN expressiveness.

Proposed method

• The authors analyze GNNs as distributed, local computation models operating on graphs, where each vertex maintains a real-valued state and exchanges messages with neighbors using learnable neural networks.
• They define a family of GNNs indexed by input size n, with bounded depth (number of message-passing rounds and neural network layers), and measure size via the number of computation nodes and absolute weight values.
• They introduce the concept of rpl-approximable activation functions (e.g., ReLU, sigmoid, tanh) and show that GNNs using such functions can simulate and be simulated by logical formulas in GFO+C.
• They use a random initialization technique (ri) to enable GNNs to compute queries in TC⁰, proving that such GNNs can simulate threshold gates and thus achieve the full power of TC⁰.
• They prove logical definability results by constructing simulations of GNNs in GFO+C and vice versa, using techniques from finite model theory, including order-invariant logic and built-in relations.
• They show that GNNs with rational weights and piecewise linear activations can compute exactly the same queries as uniform TC⁰, even without random initialization.

Experimental results

Research questions

• RQ1 What logical fragment corresponds exactly to the queries computable by bounded-depth GNNs with real weights and standard activation functions?
• RQ2 How does random initialization affect the expressiveness of GNNs, and can it extend their power to capture the complexity class TC⁰?
• RQ3 Can GNNs with only SUM aggregation and ReLU activations simulate arbitrary GNNs with more complex aggregation and activation functions, and with what computational overhead?
• RQ4 Is there a uniform characterization of GNNs that matches known uniform circuit classes like uniform TC⁰?
• RQ5 To what extent can the expressive power of GNNs be captured by first-order logic with counting, especially when restricted to rational weights and piecewise linear activations?

Key findings

• Bounded-depth GNNs with real weights and rpl-approximable activation functions compute exactly the queries definable in the guarded fragment of first-order logic with counting and built-in relations (GFO+Cₙᵤ).
• With random initialization and global readout, GNNs compute exactly the queries in the complexity class TC⁰, establishing a tight correspondence between GNNs and threshold circuits.
• GNNs using only ReLU activations and SUM aggregation can simulate arbitrary GNNs with only a polynomial size overhead, even when the original GNNs use non-linear activations and other aggregation functions.
• GNNs with rational weights and piecewise linear activations compute exactly the queries in uniform TC⁰, showing that rational weights suffice for full TC⁰ expressiveness.
• The results hold even for unbounded aggregation (sum and max), and the logical framework handles non-uniformity and built-in relations on unordered structures.
• The paper provides the first upper bound on the expressiveness of polynomial-size GNNs with random initialization, resolving a key open question in theoretical GNN research.