[Paper Review] Lyapunov-based Safe Policy Optimization for Continuous Control

Yinlam Chow, Ofir Nachum+3 more|arXiv (Cornell University)|Jan 28, 2019

Reinforcement Learning in Robotics30 references152 citations

TL;DR

The paper introduces Lyapunov-based safe policy optimization for CMDPs in continuous control, with two solvable approaches (theta-projection and a-projection) that integrate with standard policy gradients (DDPG, PPO) to guarantee safety during training and convergence, using data-efficient on/off-policy data.

ABSTRACT

We study continuous action reinforcement learning problems in which it is crucial that the agent interacts with the environment only through safe policies, i.e.,~policies that do not take the agent to undesirable situations. We formulate these problems as constrained Markov decision processes (CMDPs) and present safe policy optimization algorithms that are based on a Lyapunov approach to solve them. Our algorithms can use any standard policy gradient (PG) method, such as deep deterministic policy gradient (DDPG) or proximal policy optimization (PPO), to train a neural network policy, while guaranteeing near-constraint satisfaction for every policy update by projecting either the policy parameter or the action onto the set of feasible solutions induced by the state-dependent linearized Lyapunov constraints. Compared to the existing constrained PG algorithms, ours are more data efficient as they are able to utilize both on-policy and off-policy data. Moreover, our action-projection algorithm often leads to less conservative policy updates and allows for natural integration into an end-to-end PG training pipeline. We evaluate our algorithms and compare them with the state-of-the-art baselines on several simulated (MuJoCo) tasks, as well as a real-world indoor robot navigation problem, demonstrating their effectiveness in terms of balancing performance and constraint satisfaction. Videos of the experiments can be found in the following link: https://drive.google.com/file/d/1pzuzFqWIE710bE2U6DmS59AfRzqK2Kek/view?usp=sharing.

Motivation & Objective

• Motivate safety-critical reinforcement learning in continuous control via constrained Markov decision processes (CMDPs).
• Develop Lyapunov-based policy optimization methods that guarantee near-constraint satisfaction at every policy update.
• Enable compatibility with standard policy gradient methods (DDPG, PPO) and leverage both on-policy and off-policy data for efficiency.
• Provide two implementable approaches (theta-projection and a-projection) to handle infinite/continuous action spaces and Lyapunov constraints.

Proposed method

• Formulate safe CMDP optimization using state-dependent Lyapunov constraints to bound cumulative constraint costs.
• Introduce two solution schemes: (i) theta-projection, which optimizes policy parameters under Lyapunov constraints via projection; (ii) a-projection, which embeds the Lyapunov constraints as a safety layer projecting actions to the feasible set.
• Use Taylor-series based surrogates to convert infinite Lyapunov constraints into tractable, differentiable forms for gradient-based updates.
• Leverage on-policy (PPO) and off-policy (DDPG) algorithms to improve data efficiency and enable end-to-end training.
• Provide connections to existing safety approaches (CPO, Lagrangian) and show how Lyapunov constraints can be integrated with back-propagatable training.
• Demonstrate safe training and improved constraint satisfaction on MuJoCo benchmarks and a real-world robot navigation task.

Experimental results

Research questions

• RQ1 How can CMDPs be solved in continuous action spaces while guaranteeing safety at every policy update?
• RQ2 Can Lyapunov-based constraints be integrated with standard PG methods (PPO, DDPG) to achieve safe, data-efficient learning?
• RQ3 Do theta-projection and a-projection offer practical, scalable solutions with comparable or superior performance to existing safe RL baselines like CPO and Lagrangian methods?
• RQ4 How well do the proposed methods transfer safety guarantees from simulation to real-world robotics tasks?

Key findings

• Lyapunov-based PG algorithms maintain constraint satisfaction during training while achieving competitive performance.
• Compared to Lagrangian methods and CPO, the proposed approaches are data-efficient and can utilize both on-policy and off-policy data.
• The a-projection safety layer often yields faster convergence and less conservative updates than theta-projection, enhancing learning speed and stability.
• On MuJoCo tasks and a real Fetch robot, the methods balance performance and safety, with better generalization to new environments and transfer to real hardware.
• The framework can be implemented end-to-end and integrated with PPO or DDPG, enabling back-propagatable training without reliance on line-search or expensive backtracking.