[Paper Review] State-wise Constrained Policy Optimization

Weiye Zhao, Rui Chen+3 more|arXiv (Cornell University)|Jun 21, 2023

Reinforcement Learning in RoboticsComputer Science3 citations

TL;DR

This paper proposes State-wise Constrained Policy Optimization (SCPO), a novel policy gradient method for reinforcement learning that enforces hard, instantaneous (state-wise) safety constraints in high-dimensional robotic control. By introducing the Maximum Markov Decision Process (MMDP) framework, SCPO theoretically bounds worst-case safety violations and performance degradation, achieving state-of-the-art safety-performance trade-offs in locomotion and hazard-avoidance tasks with neural network policies.

ABSTRACT

Reinforcement Learning (RL) algorithms have shown tremendous success in simulation environments, but their application to real-world problems faces significant challenges, with safety being a major concern. In particular, enforcing state-wise constraints is essential for many challenging tasks such as autonomous driving and robot manipulation. However, existing safe RL algorithms under the framework of Constrained Markov Decision Process (CMDP) do not consider state-wise constraints. To address this gap, we propose State-wise Constrained Policy Optimization (SCPO), the first general-purpose policy search algorithm for state-wise constrained reinforcement learning. SCPO provides guarantees for state-wise constraint satisfaction in expectation. In particular, we introduce the framework of Maximum Markov Decision Process, and prove that the worst-case safety violation is bounded under SCPO. We demonstrate the effectiveness of our approach on training neural network policies for extensive robot locomotion tasks, where the agent must satisfy a variety of state-wise safety constraints. Our results show that SCPO significantly outperforms existing methods and can handle state-wise constraints in high-dimensional robotics tasks.

Motivation & Objective

• To address the lack of safety guarantees in real-world RL applications, especially for instantaneous (state-wise) constraints like collision avoidance.
• To develop a general-purpose policy optimization algorithm that ensures state-wise constraint satisfaction in expectation without requiring a known environment model.
• To provide theoretical bounds on worst-case safety violations and performance degradation during training.
• To enable effective training of deep neural network policies under strict, real-time safety constraints in high-dimensional control tasks.

Proposed method

• Introduces the Maximum Markov Decision Process (MMDP) framework to model worst-case cost accumulation across policies.
• Derives a theoretical bound on the difference between maximum costs of two policies, extending trust-region policy optimization principles to state-wise constraints.
• Designs a policy improvement step that simultaneously guarantees bounded performance degradation and enforces state-wise cost constraints.
• Approximates the theoretically justified update using a constrained optimization procedure that maintains safety during training.
• Employs a trust-region-like update with a penalty term on maximum constraint violation, ensuring conservative policy updates.
• Uses a dual-optimization approach to balance reward maximization and constraint satisfaction, with adaptive cost limits.

Experimental results

Research questions

• RQ1 Can we provide theoretical guarantees on worst-case safety violation and performance degradation when learning policies under state-wise constraints in a model-free setting?
• RQ2 How can we extend existing trust-region policy optimization frameworks to enforce hard, instantaneous constraints rather than cumulative or probabilistic ones?
• RQ3 What is the theoretical relationship between maximum cost and policy improvement under state-wise constraints?
• RQ4 Can we design a practical algorithm that achieves both high performance and strong safety guarantees in high-dimensional robotic control?
• RQ5 How does the proposed method compare to existing safe RL baselines in terms of safety, sample efficiency, and final performance on complex tasks?

Key findings

• SCPO achieves the lowest average constraint violation (ρc ≈ 0.0002) in the Drone-3DHazard-1 environment, significantly outperforming TRPO-Lagrangian (0.0007) and PCPO (0.0015).
• In the Ant-Hazard-8 task, SCPO maintains a very low maximum cost (Mc ≈ 0.0327) while achieving a high return (Jr ≈ 2.5873), outperforming CPO (Mc = 0.1330) and PCPO (Mc = 0.1046).
• On the Swimmer-Hazard-8 task, SCPO achieves the highest return (Jr = 2.6317) with the lowest constraint violation (ρc = 0.0012), surpassing TRPO (Jr = 2.6322, ρc = 0.0067) and CPO (Jr = 2.6335, ρc = 0.0045).
• SCPO consistently reduces constraint violations across all tested environments (Point-Hazard, Swimmer-Hazard, Drone-3DHazard, Ant-Hazard, Walker-Hazard) compared to all baselines.
• The algorithm demonstrates robustness in high-dimensional control tasks with thousands of policy parameters, maintaining safety without relying on prior knowledge of the dynamics model.