[Paper Review] Deep learning of dynamics and signal-noise decomposition with time-stepping constraints

Samuel Rudy, Nathan Kutz+1 more|arXiv (Cornell University)|Aug 7, 2018

Model Reduction and Neural Networks4 citations

TL;DR

This paper proposes a deep learning framework that jointly learns the dynamics and measurement noise in dynamical systems by embedding a neural network within a Runge-Kutta time-stepping structure. By enforcing time-integration constraints, the method accurately reconstructs the true vector field even from limited, noisy, or irregularly sampled data, significantly improving robustness and predictive accuracy over standard approaches.

ABSTRACT

A critical challenge in the data-driven modeling of dynamical systems is producing methods robust to measurement error, particularly when data is limited. Many leading methods either rely on denoising prior to learning or on access to large volumes of data to average over the effect of noise. We propose a novel paradigm for data-driven modeling that simultaneously learns the dynamics and estimates the measurement noise at each observation. By constraining our learning algorithm, our method explicitly accounts for measurement error in the map between observations, treating both the measurement error and the dynamics as unknowns to be identified, rather than assuming idealized noiseless trajectories. We model the unknown vector field using a deep neural network, imposing a Runge-Kutta integrator structure to isolate this vector field, even when the data has a non-uniform timestep, thus constraining and focusing the modeling effort. We demonstrate the ability of this framework to form predictive models on a variety of canonical test problems of increasing complexity and show that it is robust to substantial amounts of measurement error. We also discuss issues with the generalizability of neural network models for dynamical systems and provide open-source code for all examples.

Motivation & Objective

• To address the challenge of data-driven modeling of dynamical systems when measurements are corrupted by noise and data is limited.
• To develop a framework that simultaneously learns the underlying dynamics and estimates measurement noise without requiring prior denoising or large datasets.
• To improve model robustness and generalization by embedding the neural network within a Runge-Kutta time-integrator structure.
• To enable accurate forecasting and vector field reconstruction from sparse or non-uniformly sampled data, even with high noise levels.
• To provide a flexible, interpretable, and reproducible approach for system identification using deep learning with physical constraints.

Proposed method

• A deep neural network is used to represent the unknown vector field of a continuous dynamical system.
• The network is embedded within a classical Runge-Kutta time-integration scheme to enforce physical consistency and isolate the vector field from noise.
• The method explicitly treats both the vector field and measurement noise as unknowns to be identified during training.
• Time-stepping constraints are applied to ensure that the predicted trajectory adheres to the numerical integration scheme, even with non-uniform timesteps.
• The loss function minimizes the discrepancy between predicted and observed states while accounting for noise in the measurement process.
• The framework supports irregularly spaced data and can be trained on single long trajectories or multiple short trajectories with transients.

Experimental results

Research questions

• RQ1 Can a deep learning model jointly learn the true dynamics and measurement noise in a dynamical system without prior denoising?
• RQ2 How does enforcing a Runge-Kutta time-stepping structure improve robustness to noise and data sparsity?
• RQ3 To what extent can a single trajectory with transients enable accurate vector field reconstruction compared to multiple short trajectories?
• RQ4 How does the model generalize to regions of the state space not well-covered by training data?
• RQ5 Can this constrained learning approach outperform standard neural network models in forecasting and system identification under high noise?

Key findings

• The method achieves accurate vector field reconstruction even with 5% measurement noise, significantly outperforming models trained on single long trajectories alone.
• Training on multiple short trajectories with transients leads to a more accurate and generalizable model than training on a single long trajectory, especially in regions away from the attractor.
• The model successfully separates measurement noise from the true dynamics, enabling robust forecasting despite high noise levels.
• Neural networks with different architectures all converge to the correct limit cycle in the fluid flow example, but fail to recover the underlying radial symmetry, indicating limitations in generalization from single trajectories.
• The constrained learning framework enables reliable system identification from sparse and noisy data, with improved performance over standard black-box models.
• The open-source code (RKNN) enables reproducibility and facilitates future research in data-driven system identification.