[Paper Review] Understanding the Representation Power of Graph Neural Networks in Learning Graph Topology

Nima Dehmamy, Albert-Ĺaszló Barabási+1 more|arXiv (Cornell University)|Jul 11, 2019

Advanced Graph Neural NetworksComputer Science39 references43 citations

TL;DR

The paper analyzes how well GCNs learn graph moments that encode topology, shows limitations of standard GCNs, and proposes a modular, residual GCN design with multiple propagation rules to improve representation power, enabling discrimination of small graphs from different generators.

ABSTRACT

To deepen our understanding of graph neural networks, we investigate the representation power of Graph Convolutional Networks (GCN) through the looking glass of graph moments, a key property of graph topology encoding path of various lengths. We find that GCNs are rather restrictive in learning graph moments. Without careful design, GCNs can fail miserably even with multiple layers and nonlinear activation functions. We analyze theoretically the expressiveness of GCNs, concluding a modular GCN design, using different propagation rules with residual connections could significantly improve the performance of GCN. We demonstrate that such modular designs are capable of distinguishing graphs from different graph generation models for surprisingly small graphs, a notoriously difficult problem in network science. Our investigation suggests that, depth is much more influential than width, with deeper GCNs being more capable of learning higher order graph moments. Additionally, combining GCN modules with different propagation rules is critical to the representation power of GCNs.

Motivation & Objective

• Investigate how well GCNs learn graph moments that characterize graph topology.
• Theoretically analyze the dependence of representation power on depth versus width in GCNs.
• Develop a modular GCN design combining different propagation rules to enhance expressiveness while preserving permutation invariance.
• Demonstrate that modular GCNs with residual connections can distinguish graphs generated by different models on small graphs.
• Provide practical guidance for architecture design to improve graph topology learning with GCNs.

Proposed method

• Define graph moments and show how permutation invariance constrains learnable moments.
• Theoretically prove limitations of single-layer and multi-layer GCNs in learning higher-order graph moments without special design.
• Introduce a modular GCN design with three propagation rules and residual connections to learn polynomials of graph moments.
• Evaluate modular GCNs on tasks distinguishing Barabási–Albert, Erdős–Rényi, and configuration-model graphs on small graphs.
• Compare depth versus width effects and perform ablation studies on module combinations.

Experimental results

Research questions

• RQ1 Can GCNs learn higher-order graph moments under permutation invariance?
• RQ2 How does depth versus width affect the ability of GCNs to capture graph topology moments?
• RQ3 Does a modular design with multiple propagation rules and residual connections improve representation power beyond standard GCNs?
• RQ4 To what extent can modular GCNs distinguish graph generation models on small graphs?

Key findings

• Fully connected nets are sample- and parameter-inefficient for learning graph moments compared to GCNs.
• A single-layer GCN can learn degrees when f(A)=A, but fails for other moment forms like f(A)=D^{-1}A.
• Without residual connections, n<p layers of a GCN struggle to learn p-th order graph moments.
• With residual connections, a multi-layer GCN can learn graph moments M_p(A) with O(p) neurons, tying representation power to depth rather than graph size.
• A modular GCN with three modules and residual connections can nearly perfectly distinguish BA vs ER and BA vs Config BA graphs on small graphs.
• Depth increase has a larger impact on performance than width, and combining modules yields substantial gains in topology discrimination.