[Paper Review] Neural graphical modelling in continuous-time: consistency guarantees and algorithms

Alexis Bellot, Kim Branson+1 more|arXiv (Cornell University)|May 6, 2021

Neural Networks and Applications4 citations

TL;DR

This paper proposes a continuous-time neural graphical model (NGM) using penalized Neural Ordinary Differential Equations to learn directed graphs of local dependencies in stochastic differential equations. It establishes consistency guarantees for structure recovery using neural network parameterized vector fields and demonstrates superior performance on irregularly-sampled multivariate time series compared to state-of-the-art discrete-time methods.

ABSTRACT

The discovery of structure from time series data is a key problem in fields of study working with complex systems. Most identifiability results and learning algorithms assume the underlying dynamics to be discrete in time. Comparatively few, in contrast, explicitly define dependencies in infinitesimal intervals of time, independently of the scale of observation and of the regularity of sampling. In this paper, we consider score-based structure learning for the study of dynamical systems. We prove that for vector fields parameterized in a large class of neural networks, least squares optimization with adaptive regularization schemes consistently recovers directed graphs of local independencies in systems of stochastic differential equations. Using this insight, we propose a score-based learning algorithm based on penalized Neural Ordinary Differential Equations (modelling the mean process) that we show to be applicable to the general setting of irregularly-sampled multivariate time series and to outperform the state of the art across a range of dynamical systems.

Motivation & Objective

• To address the limitations of discrete-time graphical models in capturing true continuous-time dependencies in dynamical systems.
• To develop a score-based learning algorithm that is robust to irregular sampling and varying observation frequencies.
• To establish theoretical consistency guarantees for structure recovery using neural network parameterizations of vector fields in stochastic differential equations.
• To enable accurate and interpretable discovery of local, direct dependencies in complex systems from subsampled or irregular time series data.

Proposed method

• Formalizes the underlying dynamical system as a stochastic differential equation with a neural network parameterized drift function f(θ).
• Uses least squares optimization with adaptive group lasso regularization to estimate the vector field f(θ) and recover the adjacency matrix G.
• Applies penalized Neural Ordinary Differential Equations to model the mean process and estimate continuous-time dynamics from discrete, possibly irregular, observations.
• Employs proximal gradient descent for optimization and adaptive regularization to ensure sparsity and consistency in edge detection.
• Uses consistency arguments based on derivative approximation and regularization to prove that the method consistently recovers the true local independence graph.
• Compares NGM to discrete-time baselines like Neural Granger Causality, PCMCI, and SVAM using multiple dynamical systems and evaluation metrics.

Experimental results

Research questions

• RQ1 Can a neural network-based continuous-time model consistently recover the true underlying directed graph of local dependencies in stochastic differential equations?
• RQ2 How does the performance of continuous-time structure learning compare to discrete-time methods under irregular sampling and varying observation frequencies?
• RQ3 To what extent does adaptive regularization in a penalized Neural ODE framework improve structure recovery accuracy and robustness?
• RQ4 Does the proposed method maintain consistency and accuracy when applied to nonlinear, chaotic systems such as the Lorenz and Rössler attractors?
• RQ5 Can the method effectively learn causal structures from subsampled and noisy time series data without assuming regular sampling intervals?

Key findings

• The proposed NGM method consistently recovers the true adjacency matrix of local independencies in stochastic differential equations when the vector field is parameterized by a wide class of neural networks.
• The NGM outperforms state-of-the-art discrete-time methods such as Neural Granger Causality and PCMCI across all tested dynamical systems, including the Lorenz and Rössler systems.
• The method is robust to irregular sampling and varying observation frequencies, with significantly lower error rates than discrete-time alternatives, especially at low sampling rates.
• In the glycolysis oscillator benchmark, NGM achieved high F1 scores and accurately recovered the known nonlinear regulatory structure despite noise and nonlinearity.
• The use of adaptive group lasso regularization enables consistent edge detection without manual thresholding, with non-zero parameters directly indicating the presence of edges.
• Empirical results show that NGM maintains high true positive rate and low false discovery rate across diverse systems, even under high noise and subsampling.