[Paper Review] Reconstruction for Powerful Graph Representations

Leonardo Cotta, Christopher Morris+1 more|arXiv (Cornell University)|Oct 1, 2021

Advanced Graph Neural Networks100 references4 citations

TL;DR

This paper introduces k-Reconstruction GNNs, a novel framework that enhances graph neural networks (GNNs) by reconstructing graphs from their k-vertex subgraphs, significantly boosting expressive power while preserving permutation invariance. The method enables GNNs to solve seven previously unsolvable graph property tasks and improves performance on nine real-world benchmarks by up to 25%, demonstrating both theoretical expressiveness and practical scalability.

ABSTRACT

Graph neural networks (GNNs) have limited expressive power, failing to represent many graph classes correctly. While more expressive graph representation learning (GRL) alternatives can distinguish some of these classes, they are significantly harder to implement, may not scale well, and have not been shown to outperform well-tuned GNNs in real-world tasks. Thus, devising simple, scalable, and expressive GRL architectures that also achieve real-world improvements remains an open challenge. In this work, we show the extent to which graph reconstruction -- reconstructing a graph from its subgraphs -- can mitigate the theoretical and practical problems currently faced by GRL architectures. First, we leverage graph reconstruction to build two new classes of expressive graph representations. Secondly, we show how graph reconstruction boosts the expressive power of any GNN architecture while being a (provably) powerful inductive bias for invariances to vertex removals. Empirically, we show how reconstruction can boost GNN's expressive power -- while maintaining its invariance to permutations of the vertices -- by solving seven graph property tasks not solvable by the original GNN. Further, we demonstrate how it boosts state-of-the-art GNN's performance across nine real-world benchmark datasets.

Motivation & Objective

• Address the limited expressive power of standard GNNs, which fail to distinguish certain graph classes like cycle graphs and strongly regular graphs.
• Overcome the scalability and implementation challenges of existing expressive GNN architectures based on k-WL or isomorphism testing.
• Develop a practical, scalable, and theoretically grounded method to enhance GNNs using graph reconstruction as an inductive bias.
• Demonstrate that reconstruction improves generalization by acting as a lower-variance risk estimator under vertex-removal invariance.
• Bridge graph theory and representation learning by leveraging the reconstruction conjecture to design universal approximators without relying on expensive isomorphism tests.

Proposed method

• Propose k-Reconstruction Neural Networks (k-RNNs), a class of expressive GNNs that reconstruct a graph from its induced k-vertex subgraphs.
• Introduce k-Reconstruction GNNs as a general-purpose method to boost any existing GNN architecture by incorporating subgraph reconstruction.
• Use the reconstruction conjecture as a theoretical foundation to prove that k-Reconstruction GNNs achieve universal approximation for bounded-size attributed graphs.
• Design a differentiable, scalable training procedure that samples subgraphs during training, reducing memory and computational costs compared to full isomorphism testing.
• Leverage vertex representation changes under vertex removal as a signal for reconstructing global graph properties like diameter and spectral radius.
• Maintain permutation invariance by ensuring subgraph representations are order-agnostic through symmetric aggregation in the GNN layers.

Experimental results

Research questions

• RQ1 Can graph reconstruction from k-vertex subgraphs serve as a powerful inductive bias to enhance the expressive power of GNNs?
• RQ2 To what extent can k-Reconstruction GNNs distinguish graph classes that 1-WL and 2-WL fail to separate, such as cycle graphs and strongly regular graphs?
• RQ3 How does reconstruction improve generalization in real-world graph learning tasks, especially under vertex-removal invariance?
• RQ4 Can reconstruction be integrated into existing GNN architectures without sacrificing scalability or permutation invariance?
• RQ5 Does subgraph sampling during training preserve performance while reducing computational cost compared to exact reconstruction?

Key findings

• k-Reconstruction GNNs successfully solve seven synthetic graph property tasks—such as cycle detection and diameter estimation—that are unsolvable by standard GNNs.
• The method improves GNN performance on nine real-world benchmark datasets by up to 25%, with consistent gains across diverse tasks including molecular property prediction and social network analysis.
• Using (n-1)-Reconstruction GNNs with subgraph sampling (e.g., 15 subgraphs) achieves training loss and convergence speed within 21% of the exact model, while using fixed GPU memory regardless of graph size.
• Reconstruction acts as a lower-variance risk estimator when the underlying graph-generating distribution is invariant to vertex removals, improving generalization.
• k-Reconstruction GNNs maintain permutation invariance—unlike Ring-GNN, PPGN, and Positional GIN—while solving all tested graph property tasks.
• The approach avoids expensive isomorphism testing and fixed-size unattributed graph constraints, enabling scalable universal approximation for bounded-size attributed graphs.