[Paper Review] PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equations

Moshe Eliasof, Eldad Haber+1 more|arXiv (Cornell University)|Aug 4, 2021

Model Reduction and Neural Networks43 citations

TL;DR

PDE-GCN introduces diffusive, hyperbolic, and mixed graph convolution layers inspired by discretized PDEs on graphs to control over-smoothing and achieve competitive or state-of-the-art results across diverse tasks.

ABSTRACT

Graph neural networks are increasingly becoming the go-to approach in various fields such as computer vision, computational biology and chemistry, where data are naturally explained by graphs. However, unlike traditional convolutional neural networks, deep graph networks do not necessarily yield better performance than shallow graph networks. This behavior usually stems from the over-smoothing phenomenon. In this work, we propose a family of architectures to control this behavior by design. Our networks are motivated by numerical methods for solving Partial Differential Equations (PDEs) on manifolds, and as such, their behavior can be explained by similar analysis. Moreover, as we demonstrate using an extensive set of experiments, our PDE-motivated networks can generalize and be effective for various types of problems from different fields. Our architectures obtain better or on par with the current state-of-the-art results for problems that are typically approached using different architectures.

Motivation & Objective

• Motivate GCN design from PDE theory to understand dynamics on graphs and manifolds.
• Develop universal graph operators based on graph gradient and divergence.
• Instantiate PDEs on graphs to create diffusive and hyperbolic layers that prevent over-smoothing.
• Demonstrate PDE-GCN's effectiveness across diverse tasks (node classification, shape correspondence, PPI).
• Provide theoretical guarantees and empirical evidence of deep, stable GCNs.

Proposed method

• Define discrete gradient (G) and divergence on graphs to approximate differential operators.
• Formulate non-linear diffusion and non-linear hyperbolic PDEs on manifolds: f_t = div K* sigma(K grad f) and f_tt = div K* sigma(K grad f).
• Discretize using forward Euler (diffusion) and leapfrog (hyperbolic) schemes: f^{(l+1)} = f^{(l)} - h G^T K_l^T sigma(K_l G f^{(l)}) and f^{(l+1)} = 2 f^{(l)} - f^{(l-1)} - h^2 G^T K_l^T sigma(K_l G f^{(l)}).
• Embed inputs with 1x1 convolutions and optionally incorporate edge features via A and G to form f_0.
• Allow a mixture dynamics (PDE-GCN M) via alpha f_tt + (1-alpha) f_t = div K^T sigma(K grad f), with alpha learned.
• Initialize K_l as identity to start from a diffusion/hyperbolic prior and enable pruning via stability considerations.

Experimental results

Research questions

• RQ1 Can PDE-inspired graph dynamics (diffusion, hyperbolic, or mixed) prevent over-smoothing in deep GCNs across tasks?
• RQ2 Do PDE-GCN variants generalize across domains (3D shape classification, node classification, PPI, dense shape correspondence) better than domain-specific GCNs?
• RQ3 What is the impact of learning the mixture parameter alpha on performance for different problem types?
• RQ4 Do discretized PDE operators on graphs offer theoretical guarantees and practical stability for deep architectures?
• RQ5 How do diffusion vs. hyperbolic dynamics perform on problems emphasizing smoothing vs. conservation?

Key findings

• PDE-GCN D (diffusion) achieves high accuracy on ModelNet-10 (92.2) and Cora (83.6).
• PDE-GCN D matches or surpasses state-of-the-art on semi-supervised node classification across Cora, Citeseer, PubMed with deep layers without over-smoothing.
• PDE-GCN M (mixed dynamics) learns alpha to favor diffusion for node classification and hyperbolic dynamics for energy-conserving tasks like dense shape correspondence.
• PDE-GCN H (hyperbolic) excels in dense shape correspondence with near-perfect zero geodesic error (99.9% on FAUST).
• PDE-GCN D outperforms or matches baselines on fully supervised tasks, achieving 89.68%+ on several small datasets (Texas, Wisconsin, etc.).
• Inductive learning on PPI shows PDE-GCN D (99.07) and PDE-GCN M (99.18) competitive with or surpassing many baselines.