[Paper Review] Physics-informed Dyna-Style Model-Based Deep Reinforcement Learning for Dynamic Control

Xin‐Yang Liu, Jianxun Wang|arXiv (Cornell University)|Jul 30, 2021

Model Reduction and Neural NetworksPhysics and Astronomy89 references41 citations

TL;DR

This paper proposes a physics-informed model-based reinforcement learning framework (PiMBRL) that integrates governing physical laws—such as partial differential equations—into deep neural network dynamics models to reduce model bias and improve sample efficiency. By enforcing physical constraints during training, PiMBRL achieves higher predictive accuracy and faster convergence than model-free or purely data-driven model-based methods in chaotic systems like the Kuramoto-Sivashinsky equation.

ABSTRACT

Model-based reinforcement learning (MBRL) is believed to have much higher sample efficiency compared to model-free algorithms by learning a predictive model of the environment. However, the performance of MBRL highly relies on the quality of the learned model, which is usually built in a black-box manner and may have poor predictive accuracy outside of the data distribution. The deficiencies of the learned model may prevent the policy from being fully optimized. Although some uncertainty analysis-based remedies have been proposed to alleviate this issue, model bias still poses a great challenge for MBRL. In this work, we propose to leverage the prior knowledge of underlying physics of the environment, where the governing laws are (partially) known. In particular, we developed a physics-informed MBRL framework, where governing equations and physical constraints are utilized to inform the model learning and policy search. By incorporating the prior information of the environment, the quality of the learned model can be notably improved, while the required interactions with the environment are significantly reduced, leading to better sample efficiency and learning performance. The effectiveness and merit have been demonstrated over a handful of classic control problems, where the environments are governed by canonical ordinary/partial differential equations.

Motivation & Objective

• Address the critical challenge of model bias in model-based reinforcement learning (MBRL), especially in data-scarce and out-of-distribution regimes.
• Improve sample efficiency in training RL agents for complex physical systems, such as fluid flows governed by PDEs, where real-world interactions are costly and risky.
• Leverage known physical laws (e.g., governing PDEs) as inductive biases to constrain and regularize the learned dynamics model, reducing overfitting and improving generalization.
• Develop a unified framework that jointly optimizes the dynamics model and policy using physics-informed constraints, enabling faster and more robust learning.
• Demonstrate the superiority of the physics-informed approach over both model-free and purely data-driven MBRL in canonical chaotic control problems.

Proposed method

• Propose a physics-informed MBRL framework (PiMBRL) that embeds known governing equations (e.g., viscous Burgers’ and Kuramoto-Sivashinsky PDEs) as soft constraints during model training.
• Use an autoencoding-based recurrent neural network to learn spatiotemporal dynamics from limited environment interactions, with physics constraints enforced via a loss term.
• Introduce a model accuracy threshold λ to dynamically determine when to use the learned model for rollouts, balancing exploration and exploitation.
• Train the dynamics model and policy jointly using a twin-delayed deep deterministic policy gradient (TD3) algorithm, with the model guided by both data and physical laws.
• Incorporate physics constraints into the loss function via residual minimization of the PDE residual, ensuring the model respects underlying physical behavior.
• Use a hybrid replay buffer combining real environment transitions (Dr) and model-generated rollouts (Df), with a threshold-based switching mechanism to improve data efficiency.

Experimental results

Research questions

• RQ1 Can integrating known physical laws into the dynamics model of MBRL significantly reduce model bias and improve predictive accuracy in chaotic systems?
• RQ2 How does the physics-informed model-based approach compare to model-free and purely data-driven MBRL in terms of sample efficiency and final performance?
• RQ3 What is the impact of the model accuracy threshold λ on the learning speed and convergence of the RL agent?
• RQ4 How does the joint training of the physics-informed model and policy improve generalization and robustness in out-of-distribution regimes?
• RQ5 To what extent can physics-informed constraints reduce overfitting in data-scarce environments with high-dimensional, complex dynamics?

Key findings

• PiMBRL achieves significantly higher model predictive accuracy than purely data-driven MBRL, especially in long-horizon rollouts, due to the incorporation of physical constraints.
• The method demonstrates superior sample efficiency, with the RL agent converging faster and reaching higher total returns compared to model-free and standard MBRL baselines.
• In the Kuramoto-Sivashinsky (KS) system, PiMBRL with λ = 0.02 achieved slightly faster early convergence than λ = 0.01 or λ = 0.005, due to earlier access to model-generated data.
• All tested λ values (0.02, 0.01, 0.005) led to similar final performance, indicating that the model quality converges to a comparable level regardless of threshold, due to joint training with the policy.
• The framework effectively reduces overfitting and improves robustness in data-scarce and out-of-distribution regimes by embedding physical laws as inductive biases.
• Empirical results on viscous Burgers’ and KS equations confirm that PiMBRL outperforms both model-free and data-driven MBRL in terms of model accuracy, learning speed, and final return.