[Paper Review] Learning Continuous-Time Dynamics by Stochastic Differential Networks

Yingru Liu, Yucheng Xing+5 more|arXiv (Cornell University)|Jun 11, 2020

Neural Networks and Applications26 references4 citations

TL;DR

This paper proposes Variational Stochastic Differential Networks (VSDN), a deep learning framework that models continuous-time stochastic dynamics in irregular and sparse time series using stochastic differential equations (SDEs) and variational inference. By incorporating continuous-time VAE and IWAE losses, VSDN achieves state-of-the-art performance in prediction and interpolation tasks on sporadic data, outperforming existing models like LatentODE and ODE-RNN on both synthetic and real-world datasets including Double-OU and USHCN.

ABSTRACT

Learning continuous-time stochastic dynamics is a fundamental and essential problem in modeling sporadic time series, whose observations are irregular and sparse in both time and dimension. For a given system whose latent states and observed data are high-dimensional, it is generally impossible to derive a precise continuous-time stochastic process to describe the system behaviors. To solve the above problem, we apply Variational Bayesian method and propose a flexible continuous-time stochastic recurrent neural network named Variational Stochastic Differential Networks (VSDN), which embeds the complicated dynamics of the sporadic time series by neural Stochastic Differential Equations (SDE). VSDNs capture the stochastic dependency among latent states and observations by deep neural networks. We also incorporate two differential Evidence Lower Bounds to efficiently train the models. Through comprehensive experiments, we show that VSDNs outperform state-of-the-art continuous-time deep learning models and achieve remarkable performance on prediction and interpolation tasks for sporadic time series.

Motivation & Objective

• To address the challenge of modeling continuous-time stochastic dynamics in systems with irregular and sparse observations.
• To develop a flexible deep learning architecture capable of capturing nonlinear and stochastic dependencies in multivariate time series.
• To extend variational inference to continuous-time state-space models using VAE and IWAE objectives for efficient training.
• To improve upon existing Neural ODE-based models by incorporating latent state trajectories and stochasticity for better generalization.
• To demonstrate superior performance on prediction and interpolation tasks for sporadic time series data.

Proposed method

• Proposes Variational Stochastic Differential Networks (VSDN), a continuous-time stochastic recurrent neural network using SDEs to model latent state dynamics.
• Introduces continuous-time variants of Variational Auto-Encoding (VAE) and Importance-Weighted Auto-Encoding (IWAE) lower bounds for training.
• Employs two differential Evidence Lower Bounds (ELBOs) to efficiently optimize the model using stochastic gradient descent.
• Designs two VSDN variants: VSDN-F (filtering) and VSDN-S (smoothing), differing in inference model structure.
• Uses deep neural networks to parameterize drift and diffusion functions in the SDEs, enabling nonlinear dynamics modeling.
• Applies stochastic adjoint methods for gradient computation, enabling end-to-end training of the SDE-based architecture.

Experimental results

Research questions

• RQ1 Can a deep neural network architecture effectively model continuous-time stochastic dynamics from irregular and sparse observations?
• RQ2 How do continuous-time VAE and IWAE losses compare in optimizing SDE-based models for time series?
• RQ3 Does incorporating latent state trajectories improve modeling capacity compared to deterministic ODE-based approaches?
• RQ4 Can VSDN generalize to real-world sporadic time series with missing dimensions and irregular sampling?
• RQ5 What is the impact of using importance-weighted training objectives on model performance and uncertainty estimation?

Key findings

• VSDN-F and VSDN-S both outperform state-of-the-art models, including LatentODE, LatentSDE, GRU-ODE, and ODE-RNN, on prediction and interpolation tasks.
• On the Double-OU dataset, VSDN-S (VAE) achieves a test NLL of -1.145 ± 0.013 and MSE of 0.0065 ± 0.0002, significantly outperforming LatentODE (NLL: 0.351 ± 0.023).
• On the USHCN dataset, VSDN-S (IWAE) achieves the lowest NLL (0.654 ± 0.084) and MSE (0.381 ± 0.058), outperforming ODE-RNN and other baselines.
• The IWAE loss yields tighter bounds and better performance than VAE loss when the number of sampled trajectories is small, especially in early training stages.
• VSDN-S (VAE) requires more training epochs to converge due to a looser ELBO, indicating higher training difficulty compared to VSDN-F.
• The model demonstrates robustness on high-dimensional, sparse, and irregular data, including real-world climate and human motion sequences.