[Paper Review] The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimation

Qiang Du, Yiqi Gu+2 more|arXiv (Cornell University)|Mar 21, 2021

Model Reduction and Neural Networks54 references4 citations

TL;DR

This paper presents a rigorous convergence analysis for deep learning-based linear multistep methods (LMMs) in discovering dynamical systems. It establishes that the ℓ² grid error is bounded by O(h^p) plus network approximation error, where h is the time step and p is the local truncation order, validating high-order accuracy in numerical experiments on systems like the Lorenz equation.

ABSTRACT

Identifying hidden dynamics from observed data is a significant and challenging task in a wide range of applications. Recently, the combination of linear multistep methods (LMMs) and deep learning has been successfully employed to discover dynamics, whereas a complete convergence analysis of this approach is still under development. In this work, we consider the deep network-based LMMs for the discovery of dynamics. We put forward error estimates for these methods using the approximation property of deep networks. It indicates, for certain families of LMMs, that the $\ell^2$ grid error is bounded by the sum of $O(h^p)$ and the network approximation error, where $h$ is the time step size and $p$ is the local truncation error order. Numerical results of several physically relevant examples are provided to demonstrate our theory.

Motivation & Objective

• To develop a theoretical convergence framework for deep network-based linear multistep methods (LMMs) in data-driven dynamical system discovery.
• To quantify the ℓ² grid error in terms of time step size h and network approximation error.
• To establish conditions under which the condition number of the LMM matrix remains uniformly bounded.
• To validate the theoretical error estimates through numerical experiments on benchmark dynamical systems.
• To explore the role of implicit regularization in deep learning when auxiliary conditions are omitted in LMM formulations.

Proposed method

• Formulates the discovery of dynamics as a least-squares minimization of residual errors from LMM discretization.
• Uses deep neural networks as the ansatz space to approximate the unknown vector field.
• Derives error bounds via the approximation properties of deep networks and the stability of LMM schemes.
• Analyzes the 2-condition number of the system matrix A_h to ensure uniform boundedness across h.
• Applies the root condition for LMMs to guarantee stability and bounded condition numbers.
• Employs numerical experiments with A-B and BDF schemes to validate theoretical convergence rates.

Experimental results

Research questions

• RQ1 What is the convergence rate of deep network-based LMMs in discovering dynamical systems with respect to the time step h?
• RQ2 How does the network approximation error interact with the local truncation error of LMMs in determining the overall ℓ² grid error?
• RQ3 Under what conditions is the condition number of the LMM system matrix uniformly bounded as h → 0?
• RQ4 Can deep learning with gradient descent still yield accurate solutions when auxiliary conditions are omitted in LMM-based discovery?
• RQ5 How do the theoretical error bounds compare to empirical performance in real-world dynamical systems like the Lorenz system?

Key findings

• The ℓ² grid error of the learned dynamics is bounded by O(κ₂(A_h)(h^p + e_𝒜)), where p is the local truncation order and e_𝒜 is the network approximation error.
• For stable LMMs, the condition number κ₂(A_h) remains uniformly bounded as h → 0, ensuring numerical stability.
• Numerical results confirm convergence rates close to the theoretical O(h^p) for A-B and BDF schemes across multiple test cases.
• The error remains below O(10⁻³) across the domain Ω in 2D test problems, indicating high accuracy of the learned networks.
• Even without auxiliary conditions, gradient descent with deep networks achieves comparable error levels to stable schemes, suggesting strong implicit regularization.
• Generalization error beyond sampled grid points remains theoretically unbounded, though numerical results show excellent out-of-sample performance.