[Paper Review] Graph Neural Ordinary Differential Equations

Michael Poli, Stefano Massaroli+4 more|arXiv (Cornell University)|Nov 18, 2019

Advanced Graph Neural Networks69 references61 citations

TL;DR

GDEs provide a continuous-depth framework for graph neural networks by parameterizing node feature dynamics with graph-based ODEs, improving static and dynamic tasks over discrete GNNs. They can model sequences of graphs, handle irregular timestamps, and embed numerical ODE solvers in forward passes.

ABSTRACT

We introduce the framework of continuous--depth graph neural networks (GNNs). Graph neural ordinary differential equations (GDEs) are formalized as the counterpart to GNNs where the input-output relationship is determined by a continuum of GNN layers, blending discrete topological structures and differential equations. The proposed framework is shown to be compatible with various static and autoregressive GNN models. Results prove general effectiveness of GDEs: in static settings they offer computational advantages by incorporating numerical methods in their forward pass; in dynamic settings, on the other hand, they are shown to improve performance by exploiting the geometry of the underlying dynamics.

Motivation & Objective

• Introduce continuous-depth graph neural networks by defining graph neural ODEs (GDEs) as a continuum of GNN layers.
• Show compatibility of GDEs with static and autoregressive GNN models.
• Demonstrate computational and accuracy advantages of GDEs in static tasks and dynamic settings.
• Validate GDEs on node classification, trajectory extrapolation, and traffic forecasting.

Proposed method

• Formulate GDEs as a Cauchy problem with dot{H}(s)=F_G(H(s), Θ) and H(0)=X_e.
• Define F_G as a graph-conditioned vector field, enabling depth-variant dynamics over a graph G.
• Specialize to static models like GCDE and relate to continuous counterparts of GCNs.
• Extend GDEs to spatio-temporal settings via hybrid continuous-discrete dynamics for autoregressive graph sequences.
• Train with backpropagation, adjoint methods, or spectral-element discretization with stability considerations.
• Evaluate with fixed-step and adaptive ODE solvers (e.g., RK methods, Dormand-Prince).

Experimental results

Research questions

• RQ1 Can continuous-depth GDEs match or exceed traditional GNN performance on standard node classification benchmarks?
• RQ2 Do GCDEs leverage numerical ODE solvers to achieve better or comparable accuracy with potential depth advantages?
• RQ3 Can autoregressive GDEs model sequences of graphs with irregular timestamps and capture underlying dynamics more effectively than discrete models?
• RQ4 What are the benefits and limitations of different solver schemes (fixed vs adaptive) in static GDEs?
• RQ5 How do GDEs perform in dynamic tasks like trajectory extrapolation and traffic forecasting compared to neural ODEs and standard RNN/GRU baselines?

Key findings

• GCDE variants with higher-order solvers (rk4) can outperform standard GCN baselines on Cora, Citeseer, and Pubmed.
• Adaptive-step GCDEs provide deeper effective models but may overfit with many function evaluations (NFE).
• GCDEs show resilience to longer integration times and can mitigate node oversmoothing in static tasks.
• In multi-agent trajectory extrapolation, GCDEs (including GCDE-II) outperform Neural ODEs and plain GCDEs, leveraging relational structure.
• For traffic forecasting under irregular sampling, GCDE-GRU models outperform GRUs and GCGRUs across varying undersampling regimes.