[Paper Review] Conservative Safety Critics for Exploration

Homanga Bharadhwaj, Aviral Kumar+4 more|arXiv (Cornell University)|Oct 27, 2020

Reinforcement Learning in RoboticsComputer Science37 references32 citations

TL;DR

CSC learns a conservative safety critic to bound catastrophic failures during RL exploration, enabling safe data collection with provable failure bounds while maintaining competitive task performance.

ABSTRACT

Safe exploration presents a major challenge in reinforcement learning (RL): when active data collection requires deploying partially trained policies, we must ensure that these policies avoid catastrophically unsafe regions, while still enabling trial and error learning. In this paper, we target the problem of safe exploration in RL by learning a conservative safety estimate of environment states through a critic, and provably upper bound the likelihood of catastrophic failures at every training iteration. We theoretically characterize the tradeoff between safety and policy improvement, show that the safety constraints are likely to be satisfied with high probability during training, derive provable convergence guarantees for our approach, which is no worse asymptotically than standard RL, and demonstrate the efficacy of the proposed approach on a suite of challenging navigation, manipulation, and locomotion tasks. Empirically, we show that the proposed approach can achieve competitive task performance while incurring significantly lower catastrophic failure rates during training than prior methods. Videos are at this url https://sites.google.com/view/conservative-safety-critics/home

Motivation & Objective

• Motivate safe exploration in RL by minimizing catastrophic failures during data collection.
• Develop a conservative safety critic that overestimates failure probability to guide exploration.
• Provide theoretical guarantees bounding the probability of failures at every training iteration.
• Show that safety constraints incur limited impact on convergence and task performance.
• Demonstrate empirical effectiveness across manipulation, navigation, and locomotion tasks.

Proposed method

• Train a conservative safety critic Q_C(s,a) that upper-bounds the probability of failure using a CQL-based objective.
• Impose a KL-divergence constraint on successive policy updates to bound state distribution changes.
• Formulate a chance constraint on expected failures V_C^pi(mu) ≤ chi and solve via a primal-dual gradient method with a Lagrange multiplier.
• Use a rejection-sampling-like mechanism during environment interaction to select actions with Q_C(s,a) ≤ ε where ε adapts per iteration.
• In policy updates, replace A_C with the over-estimated or safety and solve the constrained optimization with a Fisher information-based KL approximation.
• Provide theoretical results bounding V_C^pi(mu) after updates and showing sublinear growth of cumulative safety violations under a time-varying chi.

Experimental results

Research questions

• RQ1 Can a conservative safety critic bound the probability of catastrophic failures during online RL training?
• RQ2 How does enforcing safety constraints via a conservatively learned critic affect policy learning and convergence?
• RQ3 What are the theoretical guarantees (bounds/convergence) for safety and performance under CSC?
• RQ4 Do CSC-trained policies achieve competitive task performance while reducing unsafe failures compared to prior safe exploration methods?

Key findings

• CSC reduces average catastrophic failures during training by up to 50% compared to prior safe exploration methods.
• CSC maintains competitive task reward convergence despite safety constraints.
• Theoretical results show high-probability bounds on the expected failure probability after each policy update.
• A time-varying safety threshold chi yields sublinear growth of cumulative safety violations Reg_C(T).
• Empirical evaluation across five simulated robotics domains (navigation, manipulation, locomotion) demonstrates safer exploration with CSC.