[Paper Review] Bridging Physics-based and Data-driven modeling for Learning Dynamical Systems

Rui Wang, Danielle C. Maddix+3 more|arXiv (Cornell University)|Nov 20, 2020

Model Reduction and Neural Networks37 references21 citations

TL;DR

This paper proposes AutoODE, a physics-informed deep learning framework that leverages automatic differentiation to estimate parameters in mechanistic dynamical systems, significantly outperforming data-driven models in forecasting under distribution shift. It demonstrates that deep learning models fail under data and parameter domain shifts—key challenges in real-world dynamical systems like COVID-19—while AutoODE maintains robust generalization by modeling underlying system mechanics.

ABSTRACT

How can we learn a dynamical system to make forecasts, when some variables are unobserved? For instance, in COVID-19, we want to forecast the number of infected and death cases but we do not know the count of susceptible and exposed people. While mechanics compartment models are widely used in epidemic modeling, data-driven models are emerging for disease forecasting. We first formalize the learning of physics-based models as AutoODE, which leverages automatic differentiation to estimate the model parameters. Through a benchmark study on COVID-19 forecasting, we notice that physics-based mechanistic models significantly outperform deep learning. Our method obtains a 57.4% reduction in mean absolute errors for 7-day ahead COVID-19 forecasting compared with the best deep learning competitor. Such performance differences highlight the generalization problem in dynamical system learning due to distribution shift. We identify two scenarios where distribution shift can occur: changes in data domain and changes in parameter domain (system dynamics). Through systematic experiments on several dynamical systems, we found that deep learning models fail to forecast well under both scenarios. While much research on distribution shift has focused on changes in the data domain, our work calls attention to rethink generalization for learning dynamical systems.

Motivation & Objective

• To address the challenge of forecasting dynamical systems when key variables (e.g., susceptible, exposed) are unobserved, particularly in epidemic modeling.
• To investigate why data-driven deep learning models underperform in forecasting real-world dynamical systems like COVID-19 despite strong performance on interpolation tasks.
• To identify and analyze distribution shift in both data and parameter domains as a root cause of poor generalization in deep learning for dynamical systems.
• To develop a robust, physics-informed method—AutoODE—that generalizes across unseen data and parameter regimes by leveraging automatic differentiation and mechanistic modeling.
• To benchmark physics-based models against deep learning models across multiple dynamical systems, including SEIR, Lotka-Volterra, and FitzHugh–Nagumo, under controlled distribution shifts.

Proposed method

• Formalize parameter estimation in physics-based models using automatic differentiation via numerical integration, introducing the AutoODE framework.
• Develop a novel compartmental model, ST-SuEIR, for COVID-19 forecasting that improves 7-day-ahead prediction by 57.4% in mean absolute error over the best deep learning baseline.
• Train deep learning models (Seq2Seq, FC, Transformer, NeuralODE) on synthetic time series data generated from known dynamical systems with controlled parameter and initial value variations.
• Evaluate models on both interpolation (within training domain) and extrapolation (outside training domain) test sets for data and parameter shifts.
• Use normalized system parameters and initial values to ensure fair comparison across domains, with test sets explicitly designed to test generalization under distribution shift.
• Apply RMSE as the primary metric to compare forecasting performance across models under varying distribution shift scenarios.

Experimental results

Research questions

• RQ1 Why do data-driven deep learning models fail to generalize in forecasting real-world dynamical systems like COVID-19, despite strong performance on training data?
• RQ2 How does distribution shift in the data domain (e.g., new initial conditions) affect the generalization performance of deep learning models in dynamical systems?
• RQ3 How does distribution shift in the parameter domain (e.g., changes in transmission or recovery rates) impact the forecasting accuracy of deep learning models?
• RQ4 Can physics-based models like AutoODE maintain robust performance under both data and parameter domain shifts where deep learning models fail?
• RQ5 To what extent can hybrid models that combine mechanistic understanding with deep learning improve generalization in dynamical systems forecasting?

Key findings

• Physics-based models, particularly AutoODE, significantly outperform deep learning models in forecasting cumulative confirmed, removed, and death cases in the COVID-19 epidemic, especially for infected and removed cases.
• Deep learning models (Seq2Seq, FC, Transformer, NeuralODE) exhibit high error rates—up to 7.89 RMSE—on extrapolation test sets for SEIR dynamics, while AutoODE maintains near-constant error (0.91–1.02 RMSE), indicating superior generalization.
• For Lotka-Volterra dynamics, deep learning models' RMSE on parameter extrapolation sets increased from 0.050 (interpolation) to 0.215 (extrapolation), while AutoODE's error remained stable at 0.054.
• In FitzHugh–Nagumo dynamics, the best deep learning model (Seq2Seq) had an RMSE of 4.14 on parameter extrapolation, compared to AutoODE’s 0.91, confirming its robustness under parameter shift.
• The study identifies two key sources of distribution shift—data domain and parameter domain—that severely degrade deep learning performance, even when models memorize training data.
• The ST-SuEIR model achieves a 57.4% reduction in mean absolute error for 7-day-ahead forecasting compared to the best deep learning competitor, demonstrating the advantage of physics-informed modeling in real-world epidemic forecasting.