[Paper Review] Solving multiscale dynamical systems by deep learning

Junjie Yao, Yuxiao Yi+6 more|arXiv (Cornell University)|Jan 2, 2024

Advanced Mathematical Modeling in Engineering4 citations

TL;DR

This paper introduces DeePODE, a deep learning method that combines Monte Carlo sampling with ODE-driven evolution to efficiently generate representative training data for high-dimensional multiscale dynamical systems. By leveraging the intrinsic dynamics of stiff ODEs, DeePODE overcomes the curse of dimensionality and enables deep neural networks to generalize across complex systems like turbulent flames and battery ignition, achieving high accuracy with orders-of-magnitude speedup over traditional solvers.

ABSTRACT

Multiscale dynamical systems, modeled by high-dimensional stiff ordinary differential equations (ODEs) with wide-ranging characteristic timescales, arise across diverse fields of science and engineering, but their numerical solvers often encounter severe efficiency bottlenecks. This paper introduces a novel DeePODE method, which consists of an Evolutionary Monte Carlo Sampling method (EMCS) and an efficient end-to-end deep neural network (DNN) to predict multiscale dynamical systems. We validate this finding across dynamical systems from ecological systems to reactive flows, including a predator-prey model, a power system oscillation, a battery electrolyte thermal runaway, and turbulent reaction-diffusion systems with complex chemical kinetics. The method demonstrates robust generalization capabilities, allowing pre-trained DNN models to accurately predict the behavior in previously unseen scenarios, largely due to the delicately constructed dataset. While theoretical guarantees remain to be established, empirical evidence shows that DeePODE achieves the accuracy of implicit numerical schemes while maintaining the computational efficiency of explicit schemes. This work underscores the crucial relationship between training data distribution and neural network generalization performance. This work demonstrates the potential of deep learning approaches in modeling complex dynamical systems across scientific and engineering domains.

Motivation & Objective

• To address the computational bottleneck in simulating high-dimensional stiff ODEs with widely varying timescales across scientific and engineering domains.
• To overcome the curse of dimensionality in training data generation for deep neural network (DNN) surrogates of multiscale systems.
• To develop a generalizable DNN surrogate model that maintains accuracy across unseen initial conditions and complex spatiotemporal dynamics.
• To enable efficient, stable, and accurate simulation of multiscale systems such as turbulent flames and power system oscillations using deep learning.
• To demonstrate that embedding system dynamics into the sampling process enhances data efficiency and model generalization.

Proposed method

• DeePODE employs a global multiscale sampling strategy that combines Monte Carlo sampling with brief ODE integration to enrich samples with dynamical information.
• Each randomly sampled initial state is evolved via the governing ODEs for a short time, concentrating samples in regions of slow dynamics and improving data representativeness.
• The method adaptively captures local system characteristics by using the ODE's intrinsic evolution, reducing sampling inefficiency in high-dimensional spaces.
• A deep neural network is trained on the evolved sample set to learn the state transition function, enabling fast prediction of future states.
• The approach avoids the need for extensive trajectory data by using ODE evolution to generate informative, multiscale-aware training data.
• The method is applied in conjunction with existing solvers (e.g., CVODE, Cantera) to replace reaction rate computation in specific regions, particularly where high resolution is needed.

Figure 1: Schematic diagram of the DeePODE method. (A) Flowchart of DeePODE method. After obtaining a manifold dataset and an MC dataset, each point of the MC dataset is taken as the initial condition of a corresponding dynamical system. Sample along the evolution trajectory of MC data sparsely and

Experimental results

Research questions

• RQ1 Can deep learning surrogate models trained on ODE-evolved samples generalize across diverse initial conditions and complex dynamical regimes in high-dimensional multiscale systems?
• RQ2 How can sampling strategies be designed to overcome the curse of dimensionality while preserving multiscale information in high-dimensional phase space?
• RQ3 To what extent can a DNN surrogate trained on 0D and 1D data accurately predict 2D and 3D turbulent combustion dynamics?
• RQ4 Can the integration of ODE evolution into the sampling pipeline significantly improve data efficiency and model accuracy compared to standard Monte Carlo or uniform sampling?
• RQ5 What is the computational performance gain of using DeePODE over traditional ODE solvers in complex systems like turbulent flames and battery electrolyte ignition?

Key findings

• DeePODE enables accurate simulation of turbulent flames using DNN surrogates trained only on 0D and 1D data, with predictions matching CVODE results across 2D and 3D domains.
• The method achieved stable and accurate predictions for Sandia Flame D, with radial profiles of temperature, O₂, CO₂, and velocity closely matching both CVODE and experimental data.
• For DRM19, DeePODE captured the temperature distribution of a spherical flame at 1.5 ms with high fidelity, comparable to CVODE results.
• The use of GPU accelerated reaction rate computation by over one order of magnitude, with more than two orders of magnitude speedup for larger mechanisms like GRI and n-heptane.
• The method demonstrated strong generalization to unseen conditions, including turbulent combustion and complex chemical kinetics, without retraining.
• The ODE-evolved sampling strategy effectively concentrated samples in low-dynamics regions, improving data efficiency and reducing the impact of high-dimensional concentration of measure.

Figure 2: Two-dimensional predator-prey model. DNN prediction and direct integration results for $x_{1}$ from the initial value $x_{1}=3$ , $x_{2}=2$ . MC and MF represent Monte Carlo and manifold sampling methods, respectively.