[Paper Review] Safe Model-based Reinforcement Learning with Stability Guarantees

Felix Berkenkamp, Matteo Turchetta+2 more|arXiv (Cornell University)|May 23, 2017

Adversarial Robustness in Machine Learning336 citations

TL;DR

The paper introduces SafeLyapunovLearning, a model-based RL method that uses Lyapunov stability, Gaussian process dynamics, and Lipschitz assumptions to provide high-probability safety guarantees while safely expanding the region of attraction and improving policy performance; demonstrated on a simulated inverted pendulum.

ABSTRACT

Reinforcement learning is a powerful paradigm for learning optimal policies from experimental data. However, to find optimal policies, most reinforcement learning algorithms explore all possible actions, which may be harmful for real-world systems. As a consequence, learning algorithms are rarely applied on safety-critical systems in the real world. In this paper, we present a learning algorithm that explicitly considers safety, defined in terms of stability guarantees. Specifically, we extend control-theoretic results on Lyapunov stability verification and show how to use statistical models of the dynamics to obtain high-performance control policies with provable stability certificates. Moreover, under additional regularity assumptions in terms of a Gaussian process prior, we prove that one can effectively and safely collect data in order to learn about the dynamics and thus both improve control performance and expand the safe region of the state space. In our experiments, we show how the resulting algorithm can safely optimize a neural network policy on a simulated inverted pendulum, without the pendulum ever falling down.

Motivation & Objective

• Motivate safe reinforcement learning for safety-critical systems by guaranteeing stability via Lyapunov theory.
• Develop a model-based RL framework that leverages probabilistic dynamics to provide high-probability safety certificates.
• Show how to safely collect data to expand the safe region of attraction and improve control performance.
• Provide a practical algorithm and experimental validation on a simulated inverted pendulum.
• Establish theoretical guarantees for safe exploration within the region of attraction.

Proposed method

• Represent the unknown dynamics as a known prior model plus a Lipschitz continuous error term.
• Use Gaussian process models to obtain posterior means and confidence intervals for f(x,u).
• Employ a Lyapunov function to define a region of attraction and enforce a one-step decrease condition with high probability.
• Discretize the state space to verify Lyapunov decrease on a grid and extend results to the continuous space via Lipschitz continuity.
• Optimize the policy to maximize the estimated region of attraction subject to safety constraints derived from Theorem 2.
• Propose a data collection strategy that safely explores within the current safe set by prioritizing informative, safe state-action pairs (Equation 6).
• Provide a practical algorithm (Algorithm 1 SafeLyapunovLearning) that updates the policy and GP with safe measurements while guaranteeing safety within the region of attraction.

Experimental results

Research questions

• RQ1 Can a model-based RL method provide high-probability safety guarantees expressed as Lyapunov-based stability certificates?
• RQ2 How can one safely expand the region of attraction while learning unknown dynamics modeled by Gaussian processes?
• RQ3 What are the theoretical conditions under which safe exploration and data collection expand the safe set without leaving it?
• RQ4 How can Lyapunov-based safety be integrated into practical policy optimization for continuous state-action spaces?
• RQ5 Does the approach scale to nonlinear policies such as neural networks and perform safely on benchmark tasks like inverted pendulum?

Key findings

• The algorithm yields high-probability safety guarantees for stability by using Lyapunov decrease conditions with GP-based dynamics.
• Under RKHS-bounded model error and Lipschitz assumptions, safe data collection within the current region of attraction is possible and safe exploration expands the safe region.
• Theoretical results show that discretized verification suffices to guarantee continuous-space Lyapunov decrease with increasing data, linking discretization to accuracy.
• The method can safely optimize a neural network policy on a simulated inverted pendulum without the pendulum falling.
• An explicit exploration strategy prioritizes the most uncertain safe state-action pairs to efficiently enlarge the region of attraction.