[Paper Review] Safe Reinforcement Learning Using Advantage-Based Intervention

Nolan Wagener, Boots, Byron+1 more|arXiv (Cornell University)|Jun 16, 2021

Reinforcement Learning in Robotics5 citations

TL;DR

SAILR is a safe reinforcement learning algorithm that uses advantage-based intervention to ensure safety during training in unknown Markov decision processes. By leveraging off-the-shelf RL algorithms and transforming data to penalize interventions, SAILR achieves strong safety guarantees during both training and deployment, significantly reducing constraint violations compared to prior methods.

ABSTRACT

Many sequential decision problems involve finding a policy that maximizes total reward while obeying safety constraints. Although much recent research has focused on the development of safe reinforcement learning (RL) algorithms that produce a safe policy after training, ensuring safety during training as well remains an open problem. A fundamental challenge is performing exploration while still satisfying constraints in an unknown Markov decision process (MDP). In this work, we address this problem for the chance-constrained setting. We propose a new algorithm, SAILR, that uses an intervention mechanism based on advantage functions to keep the agent safe throughout training and optimizes the agent's policy using off-the-shelf RL algorithms designed for unconstrained MDPs. Our method comes with strong guarantees on safety during both training and deployment (i.e., after training and without the intervention mechanism) and policy performance compared to the optimal safety-constrained policy. In our experiments, we show that SAILR violates constraints far less during training than standard safe RL and constrained MDP approaches and converges to a well-performing policy that can be deployed safely without intervention. Our code is available at https://github.com/nolanwagener/safe_rl.

Motivation & Objective

• To address the challenge of ensuring safety during exploration in unknown MDPs, where standard safe RL methods lack training-time safety guarantees.
• To develop a method that maintains safety throughout training while enabling high policy performance, without relying on complex constrained optimization or control-theoretic assumptions.
• To provide theoretical guarantees on safety during both training and deployment, even when using standard unconstrained RL algorithms.
• To reduce the number of safety violations during training compared to existing CMDP-based and intervention-based approaches.
• To enable stable and reliable learning by using advantage functions to guide intervention decisions, making the method generic and scalable.

Proposed method

• SAILR uses an advantage-based intervention rule that triggers a backup policy when the agent's action has low advantage (i.e., is unsafe relative to a safe policy) in terms of expected return.
• During rollouts, whenever the agent proposes a high-risk action, the backup policy intervenes and guides the agent to a safe state, simulating a transition to an absorbing state with a negative reward.
• Collected trajectories are transformed into experiences for an unconstrained MDP by penalizing any state-action pairs that triggered intervention, using a surrogate MDP $ ilde{ ho}$.
• The transformed data is then used to train the agent's policy via any standard off-the-shelf RL algorithm, such as SAC or PPO.
• The method relies on mild assumptions: unsafe states are absorbing, and the backup policy ensures safety from the initial state with high probability.
• The intervention mechanism is partial and generic, depending only on estimated advantage functions, avoiding assumptions on smoothness or ergodicity.

Experimental results

Research questions

• RQ1 Can we design a safe RL algorithm that guarantees safety during both training and deployment without relying on complex constrained optimization?
• RQ2 How can we use advantage functions to guide safe interventions in a way that maintains policy performance and stability?
• RQ3 What is the impact of intervention penalty magnitude on training-time safety and final policy performance?
• RQ4 How does model bias in the intervention mechanism affect safety and learning stability?
• RQ5 Can we achieve strong safety guarantees while using standard, unconstrained RL algorithms for policy optimization?

Key findings

• SAILR reduces safety violations during training by orders of magnitude compared to standard safe RL and constrained MDP methods.
• The method achieves high deployment performance with minimal constraint violations, even when using a biased model for intervention.
• Shaping the cost function for intervention significantly reduces safety violations and stabilizes policy optimization, especially under model bias.
• Larger intervention penalties lead to faster learning of safe behaviors and improved deployment safety, consistent with theoretical bounds.
• The MPC-based intervention rule is partial—requiring only a nonzero penalty—while heuristic interventions require larger penalties to achieve similar safety, indicating greater robustness of the advantage-based approach.
• SAILR's theoretical guarantees hold under mild assumptions: absorbing unsafe states and a safe backup policy from the initial state.