[Paper Review] A Survey on Physics Informed Reinforcement Learning: Review and Open Problems

Chayan Banerjee, Kien Nguyen+2 more|arXiv (Cornell University)|Sep 5, 2023

Muscle activation and electromyography studies4 citations

TL;DR

This paper presents a comprehensive survey on Physics-Informed Reinforcement Learning (PIRL), introducing a novel taxonomy that classifies PIRL methods based on physics prior representation, integration strategies, and learning biases. It reviews state-of-the-art approaches, identifies key challenges in sample efficiency, safety, and benchmarking, and outlines open problems to guide future research in making RL more data-efficient, physically plausible, and applicable to real-world systems.

ABSTRACT

The inclusion of physical information in machine learning frameworks has revolutionized many application areas. This involves enhancing the learning process by incorporating physical constraints and adhering to physical laws. In this work we explore their utility for reinforcement learning applications. We present a thorough review of the literature on incorporating physics information, as known as physics priors, in reinforcement learning approaches, commonly referred to as physics-informed reinforcement learning (PIRL). We introduce a novel taxonomy with the reinforcement learning pipeline as the backbone to classify existing works, compare and contrast them, and derive crucial insights. Existing works are analyzed with regard to the representation/ form of the governing physics modeled for integration, their specific contribution to the typical reinforcement learning architecture, and their connection to the underlying reinforcement learning pipeline stages. We also identify core learning architectures and physics incorporation biases (i.e., observational, inductive and learning) of existing PIRL approaches and use them to further categorize the works for better understanding and adaptation. By providing a comprehensive perspective on the implementation of the physics-informed capability, the taxonomy presents a cohesive approach to PIRL. It identifies the areas where this approach has been applied, as well as the gaps and opportunities that exist. Additionally, the taxonomy sheds light on unresolved issues and challenges, which can guide future research. This nascent field holds great potential for enhancing reinforcement learning algorithms by increasing their physical plausibility, precision, data efficiency, and applicability in real-world scenarios.

Motivation & Objective

• To address the growing need for integrating physical laws into reinforcement learning to improve sample efficiency, generalization, and real-world applicability.
• To identify and systematize the diverse ways physics priors are incorporated into RL pipelines, including observational, inductive, and learning-based biases.
• To provide a unified taxonomy for classifying PIRL methods based on physics representation, integration strategy, and learning architecture.
• To highlight unresolved challenges such as high-dimensional state spaces, safe exploration in uncertain environments, and the lack of standardized benchmarks.
• To guide future research by identifying open problems and opportunities in PIRL, especially in model-agnostic safety and generalized physics-informed learning.

Proposed method

• Proposes a novel taxonomy using the reinforcement learning pipeline as a backbone to classify PIRL methods along three axes: physics prior type, representation form, and integration strategy.
• Categorizes PIRL approaches based on three physics incorporation biases: observational (e.g., using physical constraints as supervision), inductive (e.g., embedding physics into network architecture), and learning-based (e.g., training with physics-consistent loss functions).
• Reviews state-of-the-art PIRL methods using unified notation and functional diagrams to compare their architecture, physics integration, and training procedures.
• Analyzes the use of physics-informed world models, reward functions, and barrier certificates (e.g., data-driven CBFs) to improve simulation-to-reality transfer and safe exploration.
• Evaluates existing benchmarks and training environments used in PIRL, including custom simulators, PyBullet, MATLAB-Simulink, and MOCAP-based platforms.
• Identifies gaps in evaluation by highlighting the lack of standardized, open-source benchmarks for fair comparison across PIRL algorithms.

Figure 2 : Agent-environment framework, of RL paradigm. Here the reward generating function and the system/ plant is abstracted as the environment. And the control policy (e.g. a DNN) and the learning algorithm, forms the RL agent.

Experimental results

Research questions

• RQ1 How can physics priors be systematically classified and integrated into reinforcement learning pipelines to improve learning efficiency and physical plausibility?
• RQ2 What are the dominant strategies for incorporating physics information—observational, inductive, or learning-based—and how do they differ in effectiveness and implementation?
• RQ3 What are the key challenges in applying PIRL to high-dimensional, continuous control tasks, and how can physics-guided representation learning mitigate them?
• RQ4 How can physics-informed methods ensure safe exploration in complex and uncertain environments, especially when system models are imperfect?
• RQ5 Why is there a lack of standardized benchmarks in PIRL, and how can unified evaluation platforms accelerate progress in the field?

Key findings

• The number of PIRL papers has grown exponentially over the past six years, indicating a rising research trend and increasing interest in physics-informed methods.
• Physics-informed world models and reward functions significantly improve sample efficiency and simulation-to-reality transfer, reducing the need for costly real-world training.
• Data-driven barrier certificates based on physical constraints enable safer exploration, but their generalization across tasks remains limited.
• Representation learning in high-dimensional spaces benefits from physics-guided feature extraction, but learning physically meaningful low-dimensional representations remains an open challenge.
• The absence of standardized benchmarks and evaluation platforms hinders fair comparison and reproducibility of PIRL methods across different domains.
• Current PIRL approaches are highly task-specific and require significant domain expertise, underscoring the need for generalized, model-agnostic frameworks for physics integration.

Figure 3 : Typical RL architectures, based on model use and interaction with the environment.