[Paper Review] A Review of Safe Reinforcement Learning: Methods, Theory and Applications

Shangding Gu, Long Yang+5 more|arXiv (Cornell University)|May 20, 2022

Safety Systems Engineering in AutonomyEngineering101 citations

TL;DR

This paper surveys safe reinforcement learning, introduces the 2H3W framework (Safety Policy, Safety Complexity, Safety Applications, Safety Benchmarks, Safety Challenges), analyzes model-based and model-free methods, and discusses theory, benchmarks, and real-world applications.

ABSTRACT

Reinforcement Learning (RL) has achieved tremendous success in many complex decision-making tasks. However, safety concerns are raised during deploying RL in real-world applications, leading to a growing demand for safe RL algorithms, such as in autonomous driving and robotics scenarios. While safe control has a long history, the study of safe RL algorithms is still in the early stages. To establish a good foundation for future safe RL research, in this paper, we provide a review of safe RL from the perspectives of methods, theories, and applications. Firstly, we review the progress of safe RL from five dimensions and come up with five crucial problems for safe RL being deployed in real-world applications, coined as "2H3W". Secondly, we analyze the algorithm and theory progress from the perspectives of answering the "2H3W" problems. Particularly, the sample complexity of safe RL algorithms is reviewed and discussed, followed by an introduction to the applications and benchmarks of safe RL algorithms. Finally, we open the discussion of the challenging problems in safe RL, hoping to inspire future research on this thread. To advance the study of safe RL algorithms, we release an open-sourced repository containing the implementations of major safe RL algorithms at the link: https://github.com/chauncygu/Safe-Reinforcement-Learning-Baselines.git.

Motivation & Objective

• Define safety concepts in RL and relate them to existing definitions.
• Identify five core safe-RL problems (2H3W) and their implications for real-world deployment.
• Survey safe model-based and model-free algorithms with theoretical and empirical insights.
• Discuss safety benchmarks, applications, and challenges to guide future research.
• Provide an open benchmark suite and open-source implementations to support the field.

Proposed method

• Frame safe RL as Constrained Markov Decision Processes (CMDP).
• Survey model-based safe RL approaches (Lyapunov, MPC, Gaussian processes, formal methods) and their convergence analyses.
• Survey model-free safe RL approaches (policy and value-based), including CPO and primal-dual methods.
• Discuss sample complexity and convergence for safe RL methods in both tabular/linear and deep settings.
• Present safety benchmarks (e.g., AI Safety Gridworlds, Safety Gym, Safe MAMuJoCo) and the design of cost/reward functions for safety.
• Offer an open-source benchmark suite and tutorials at the provided GitHub repository link.

Experimental results

Research questions

• RQ1 What constitutes a safe policy in RL across different safety definitions?
• RQ2 How can we formulate and solve CMDP-safe RL problems with theoretical guarantees?
• RQ3 What is the sample complexity of safe RL methods in practice, especially for deep, high-dimensional problems?
• RQ4 What benchmarks fairly evaluate safety performance alongside reward optimization?
• RQ5 What are the main challenges and open questions for safe RL in real-world, multi-agent, and adversarial settings?

Key findings

• The paper proposes the 2H3W framework to structure Safe RL research around Safety Policy, Safety Complexity, Safety Applications, Safety Benchmarks, and Safety Challenges.
• It analyzes safe RL through model-based and model-free approaches, including Lyapunov-based and Gaussian-process-based methods, and discusses their convergence properties.
• It surveys and categorizes CMDP-based methods (primal-dual, CVaR, constrained policy optimization) and highlights their trade-offs in safety guarantees and computational cost.
• It reviews several real-world applications (autonomous driving, robotics, video compression) and multiple safety benchmarks (AI Safety Gridworlds, Safety Gym, Safe MAMuJoCo, Safe MARobosuite).
• The authors release a benchmark suite and open-source implementations to facilitate reproducibility and progress in safe RL research.