[Paper Review] Safe and Efficient Reinforcement Learning Using Disturbance-Observer-Based Control Barrier Functions

Yikun Cheng, Zhao Pan+1 more|arXiv (Cornell University)|Nov 30, 2022

Reinforcement Learning in Robotics5 citations

TL;DR

This paper proposes a model-free safe reinforcement learning framework using disturbance observer-based control barrier functions (DOB-CBF) to ensure safety without requiring uncertainty model learning. By estimating disturbances in real time and enforcing robust CBF conditions via quadratic programming, the method achieves significantly lower safety violation rates and higher sample and computational efficiency compared to GP-based CBF methods, as validated on a unicycle and 2D quadrotor.

ABSTRACT

Safe reinforcement learning (RL) with assured satisfaction of hard state constraints during training has recently received a lot of attention. Safety filters, e.g., based on control barrier functions (CBFs), provide a promising way for safe RL via modifying the unsafe actions of an RL agent on the fly. Existing safety filter-based approaches typically involve learning of uncertain dynamics and quantifying the learned model error, which leads to conservative filters before a large amount of data is collected to learn a good model, thereby preventing efficient exploration. This paper presents a method for safe and efficient RL using disturbance observers (DOBs) and control barrier functions (CBFs). Unlike most existing safe RL methods that deal with hard state constraints, our method does not involve model learning, and leverages DOBs to accurately estimate the pointwise value of the uncertainty, which is then incorporated into a robust CBF condition to generate safe actions. The DOB-based CBF can be used as a safety filter with model-free RL algorithms by minimally modifying the actions of an RL agent whenever necessary to ensure safety throughout the learning process. Simulation results on a unicycle and a 2D quadrotor demonstrate that the proposed method outperforms a state-of-the-art safe RL algorithm using CBFs and Gaussian processes-based model learning, in terms of safety violation rate, and sample and computational efficiency.

Motivation & Objective

• Address the inefficiency and conservatism of existing safe RL methods that rely on model learning and Gaussian process regression for uncertainty quantification.
• Enable safe exploration in model-free RL by eliminating the need for dynamic model learning while maintaining hard safety constraints.
• Improve sample and computational efficiency by replacing computationally heavy Gaussian process models with disturbance observers.
• Ensure robust safety through accurate, real-time disturbance estimation and a pre-computed estimation error bound.
• Demonstrate superior safety and efficiency compared to state-of-the-art CBF-based safe RL with GP-based uncertainty modeling.

Proposed method

• Leverages a disturbance observer (DOB) to estimate the pointwise value of the lumped uncertainty (unmodeled dynamics) in real time.
• Uses a pre-computable estimation error bound (EEB) to quantify the accuracy of the DOB's disturbance estimate.
• Integrates the estimated disturbance and EEB into a robust control barrier function (CBF) condition to enforce safety.
• Applies a quadratic program (QP) to compute minimal action corrections that satisfy the robust CBF condition, ensuring safety with minimal policy deviation.
• Operates as a safety filter that can be seamlessly integrated with any model-free RL algorithm without modifying the policy training process.
• Relies only on a known nominal model and does not require learning the true system dynamics or uncertainty distribution.

Experimental results

Research questions

• RQ1 Can a disturbance observer-based CBF framework achieve safe exploration in model-free RL without requiring dynamic model learning?
• RQ2 How does the DOB-CBF method compare to GP-based CBF methods in terms of safety violation rates during training?
• RQ3 What is the computational and sample efficiency of DOB-CBF-RL compared to GP-CBF-RL in high-dimensional robotic control tasks?
• RQ4 How stable and accurate is the disturbance estimation provided by the DOB across different training stages?
• RQ5 To what extent does the DOB-CBF framework enable more aggressive and efficient policy learning while maintaining safety?

Key findings

• The DOB-CBF-RL framework achieved a safety violation rate of 0.0% in the final 1500–2000 training episodes on the 2D quadrotor, compared to 40.4% for GP-CBF-RL.
• In the initial training phase (episodes 1–500), DOB-CBF-RL had a safety violation rate of 15.8%, while GP-CBF-RL had 91.4%, demonstrating significantly improved early safety.
• With pre-training, GP-CBF-RL reduced its violation rate to 30.8% in the first 500 episodes, but still underperformed DOB-CBF-RL, which achieved 15.8%.
• The DOB provided stable disturbance estimation with an error below 5% from the start, while the GP model required over 600,000 steps to reduce error below 5%.
• The average computation time per 1,000 steps for DOB-CBF-RL was at least three times faster than for GP-CBF-RL, highlighting superior computational efficiency.
• The DOB-CBF-RL agent learned a high-performance policy in 6000 episodes, while the GP-CBF-RL agent failed to converge to a comparable policy in the same number of episodes, indicating better sample efficiency.