[Paper Review] End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks

Richard Cheng, Gábor Orosz+2 more|arXiv (Cornell University)|Mar 21, 2019

Autonomous Vehicle Technology and Safety28 references43 citations

TL;DR

The paper presents RL-CBF, a framework that combines model-free RL with model-based control barrier functions (CBFs) and online dynamics learning via Gaussian Processes to guarantee safety during learning and improve sample efficiency in nonlinear control tasks.

ABSTRACT

Reinforcement Learning (RL) algorithms have found limited success beyond simulated applications, and one main reason is the absence of safety guarantees during the learning process. Real world systems would realistically fail or break before an optimal controller can be learned. To address this issue, we propose a controller architecture that combines (1) a model-free RL-based controller with (2) model-based controllers utilizing control barrier functions (CBFs) and (3) on-line learning of the unknown system dynamics, in order to ensure safety during learning. Our general framework leverages the success of RL algorithms to learn high-performance controllers, while the CBF-based controllers both guarantee safety and guide the learning process by constraining the set of explorable polices. We utilize Gaussian Processes (GPs) to model the system dynamics and its uncertainties. Our novel controller synthesis algorithm, RL-CBF, guarantees safety with high probability during the learning process, regardless of the RL algorithm used, and demonstrates greater policy exploration efficiency. We test our algorithm on (1) control of an inverted pendulum and (2) autonomous car-following with wireless vehicle-to-vehicle communication, and show that our algorithm attains much greater sample efficiency in learning than other state-of-the-art algorithms and maintains safety during the entire learning process.

Motivation & Objective

• Motivate safe exploration in reinforcement learning for real-world, safety-critical continuous control tasks.
• Develop a framework that guarantees safety during learning by combining model-free RL with control barrier functions (CBFs) and online dynamics learning.
• Improve exploration efficiency and sample efficiency by constraining the explored policy space with CBFs and learning dynamics online.

Proposed method

• Use Gaussian Processes to model unknown dynamics d(s) and obtain high-probability confidence intervals (mu_d, sigma_d).
• Define a safe set C via a linear barrier function h(s) and enforce forward invariance using discrete-time CBFs, formulated as a quadratic program (QP).
• Integrate a model-free RL controller u_RL with a CBF controller to create a safe, end-to-end controller via a projection-like QP (u = u_RL + u_CBF).
• Extend to CBF-guided exploration by accumulating prior CBF corrections into a guiding term u_bar that shifts the RL update toward the safe region, and solve a combined QP to obtain the deployed action.
• Provide theoretical safety guarantees: if the QP has a zero slack (epsilon=0) the safe set is forward invariant with probability 1-δ; with bounded slack, safety extends to an enlarged set C_ε with probability 1-δ.
• Offer a computationally efficient implementation by approximating the sum of past CBF terms with a neural network to reduce online complexity.

Experimental results

Research questions

• RQ1 Can model-free RL algorithms be made safe during learning by using model-based control barrier functions (CBFs)?
• RQ2 Does online learning of dynamics via Gaussian Processes enable reliable safety guarantees and adaptive conservatism in the barrier controller?
• RQ3 Does guiding policy exploration with CBFs improve sample efficiency compared to standard model-free RL in nonlinear control tasks?
• RQ4 Is it feasible to integrate RL with CBFs in a way that preserves safety while achieving competitive or superior performance compared to baseline RL methods?
• RQ5 What are the practical benefits and limits of the RL-CBF approach on real-style tasks like inverted pendulum control and vehicle-following?

Key findings

• RL-CBF achieves faster learning and higher sample efficiency than TRPO or DDPG baselines in the evaluated tasks.
• The RL-CBF framework maintains safety throughout learning by keeping the system within the safe set C (with probabilistic guarantees).
• In experiments, TRPO-CBF and DDPG-CBF converge rapidly to high-performance controllers and avoid unsafe excursions that standard RL methods exhibit during learning.
• The CBF component quickly becomes inactive as the guided RL controller learns a safe policy, indicating effective reduction of safety intervention over time.
• A practical extension using a bar-capped neural network to approximate past CBF contributions preserves safety guarantees while reducing online computation.
• Compared to baselines, the inverted pendulum task shows maintained safety and superior learning speed; the car-following task demonstrates safe, improved policy search with CBF guidance.