[Paper Review] Safe Exploration in Continuous Action Spaces

Gal Dalal, Krishnamurthy Dvijotham+4 more|arXiv (Cornell University)|Jan 26, 2018

Reinforcement Learning in Robotics16 references274 citations

TL;DR

Introduces a state-wise safety layer that analytically corrects actions to satisfy per-state safety constraints during learning, enabling zero-violation exploration in continuous action spaces using a linearized single-step safety model.

ABSTRACT

We address the problem of deploying a reinforcement learning (RL) agent on a physical system such as a datacenter cooling unit or robot, where critical constraints must never be violated. We show how to exploit the typically smooth dynamics of these systems and enable RL algorithms to never violate constraints during learning. Our technique is to directly add to the policy a safety layer that analytically solves an action correction formulation per each state. The novelty of obtaining an elegant closed-form solution is attained due to a linearized model, learned on past trajectories consisting of arbitrary actions. This is to mimic the real-world circumstances where data logs were generated with a behavior policy that is implausible to describe mathematically; such cases render the known safety-aware off-policy methods inapplicable. We demonstrate the efficacy of our approach on new representative physics-based environments, and prevail where reward shaping fails by maintaining zero constraint violations.

Motivation & Objective

• Ensure zero-per-state constraint violations during RL training in physical systems.
• Leverage single-step dynamics to pre-train a linear safety model from logged data.
• Attach a differentiable safety layer to any continuous-control policy to enforce safety.
• Demonstrate that safety-aware learning can improve convergence and performance over reward shaping.

Proposed method

• Train a state-conditioned linear safety model c̄i(s′) ≈ c̄i(s) + g(s;w_i)ᵀ a using one-step transitions from random-action logs (Equation 2).
• Pre-train g(s;w_i) on a dataset D of (s,a,s′) tuples to predict safety signal changes.
• Attach a safety layer on top of the policy that solves a constrained projection: minimize ||a−μθ(s)||² subject to c̄i(s) + g(s;w_i)ᵀ a ≤ Ci (Equation 4).
• Obtain a closed-form solution under a single-active-constraint assumption: a* = μθ(s) − λ*i* g(s;w_i*) (Equations 5–6).
• Use this layer with Deep Deterministic Policy Gradient (DDPG) or any continuous-control algorithm; the layer is differentiable and lightweight.

Experimental results

Research questions

• RQ1 Can per-state safety constraints be guaranteed during learning in continuous action spaces without assuming a known behavior policy?
• RQ2 Does a linear single-step safety model learned from random logs suffice to enforce safety through a closed-form action correction?
• RQ3 How does safety-layered learning compare to reward shaping in terms of safety guarantees and convergence efficiency?
• RQ4 Is the approach applicable across multiple physics-based domains with different safety signals and constraints?
• RQ5 What is the impact of the single-active-constraint assumption on performance and safety in practice?

Key findings

• The safety layer enforces zero constraint violations across all seeds and tasks.
• The approach accelerates convergence compared to learning without the safety layer and compared to reward shaping.
• With the safety layer, DDPG maintains safety while achieving higher discounted rewards in tested domains.
• Reward shaping fails to guarantee zero violations and can hinder convergence, whereas the safety layer guarantees safety and improves learning speed.
• The method is robot-agnostic and can be added to any continuous-control algorithm, not limited to a specific RL method.