[Paper Review] SafeRL-Kit: Evaluating Efficient Reinforcement Learning Methods for Safe Autonomous Driving

Linrui Zhang, Qin Zhang+3 more|arXiv (Cornell University)|Jun 17, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper introduces SafeRL-Kit, a unified off-policy reinforcement learning toolkit for safe autonomous driving, featuring state-of-the-art safe RL algorithms including a novel first-order method, Exact Penalty Optimization (EPO). EPO achieves robust, zero-cost-constraint performance with stable training and superior sample efficiency, outperforming existing methods in benchmark environments like SpeedLimit and MetaDrive.

ABSTRACT

Safe reinforcement learning (RL) has achieved significant success on risk-sensitive tasks and shown promise in autonomous driving (AD) as well. Considering the distinctiveness of this community, efficient and reproducible baselines are still lacking for safe AD. In this paper, we release SafeRL-Kit to benchmark safe RL methods for AD-oriented tasks. Concretely, SafeRL-Kit contains several latest algorithms specific to zero-constraint-violation tasks, including Safety Layer, Recovery RL, off-policy Lagrangian method, and Feasible Actor-Critic. In addition to existing approaches, we propose a novel first-order method named Exact Penalty Optimization (EPO) and sufficiently demonstrate its capability in safe AD. All algorithms in SafeRL-Kit are implemented (i) under the off-policy setting, which improves sample efficiency and can better leverage past logs; (ii) with a unified learning framework, providing off-the-shelf interfaces for researchers to incorporate their domain-specific knowledge into fundamental safe RL methods. Conclusively, we conduct a comparative evaluation of the above algorithms in SafeRL-Kit and shed light on their efficacy for safe autonomous driving. The source code is available at \href{ https://github.com/zlr20/saferl_kit}{this https URL}.

Motivation & Objective

• To address the lack of efficient, reproducible, and up-to-date safe RL baselines tailored for safety-critical autonomous driving tasks.
• To provide a unified, off-policy framework that enables sample-efficient learning from historical data and human demonstrations.
• To benchmark the latest safe RL algorithms—including novel methods—on realistic autonomous driving environments to identify high-performing approaches.
• To introduce and validate a new first-order optimization method, Exact Penalty Optimization (EPO), for safe RL in autonomous driving.
• To offer extensible, modular interfaces that allow researchers to integrate domain-specific knowledge into core safe RL algorithms.

Proposed method

• SafeRL-Kit implements multiple state-of-the-art safe RL algorithms under an off-policy setting to improve sample efficiency and enable reuse of past experience.
• The toolkit integrates five existing methods: Safety Layer, Recovery RL, Off-policy Lagrangian, Feasible Actor-Critic (FAC), and a novel method, Exact Penalty Optimization (EPO).
• EPO reformulates the constrained optimization problem into an equivalent unconstrained one using a single fixed penalty factor and a ReLU operator, avoiding state-dependent Lagrangian multipliers.
• All algorithms are built on a unified training framework with off-the-shelf interfaces, enabling modular extension and integration of domain-specific knowledge.
• The framework supports both vector-observation inputs (e.g., ego-state, LiDAR) and is designed for compatibility with modern deep learning libraries.
• The toolkit is released with code and is compatible with offline data, including logged trajectories and human demonstrations, enhancing data efficiency for autonomous driving.

Experimental results

Research questions

• RQ1 How do modern off-policy safe RL algorithms compare in terms of safety, sample efficiency, and convergence stability on autonomous driving benchmarks?
• RQ2 Can a first-order penalty-based method like EPO achieve comparable or better performance than primal-dual Lagrangian methods while avoiding hyperparameter sensitivity?
• RQ3 What is the impact of hyperparameter choices—particularly learning rates and penalty factors—on the training stability and performance of safe RL algorithms?
• RQ4 To what extent can off-policy safe RL methods leverage offline data and human demonstrations in autonomous driving environments?
• RQ5 How does the unified framework in SafeRL-Kit support extensibility and integration of domain-specific safety constraints?

Key findings

• EPO achieves near-zero cost return in the SpeedLimit environment and converges quickly, outperforming other methods in both safety and learning speed.
• Off-policy Lagrangian and Feasible Actor-Critic show significantly lower cumulative costs than Safety Layer and Recovery RL, indicating better constraint adherence.
• Lagrangian-based methods exhibit high sensitivity to the learning rate of the Lagrangian multiplier, with oscillations and performance degradation when tuned improperly.
• EPO is robust to penalty factor choices: performance remains stable and consistent when κ > 5, suggesting it is easier to tune than primal-dual methods.
• SafeRL-Kit reduces sample complexity by up to 10x compared to on-policy baselines, requiring only 500k–1M interactions instead of 10M for convergence in similar tasks.
• The unified framework enables seamless integration of domain-specific knowledge, and the toolkit is production-ready for deployment in data-rich autonomous driving scenarios.