[Paper Review] Responsive Safety in Reinforcement Learning by PID Lagrangian Methods

Adam Stooke, Joshua Achiam+1 more|arXiv (Cornell University)|Jul 8, 2020

Advanced Control Systems Optimization34 references44 citations

TL;DR

The paper introduces PID-based updates to the Lagrange multiplier in constrained reinforcement learning, improving safety constraint satisfaction and robustness in deep RL.

ABSTRACT

Lagrangian methods are widely used algorithms for constrained optimization problems, but their learning dynamics exhibit oscillations and overshoot which, when applied to safe reinforcement learning, leads to constraint-violating behavior during agent training. We address this shortcoming by proposing a novel Lagrange multiplier update method that utilizes derivatives of the constraint function. We take a controls perspective, wherein the traditional Lagrange multiplier update behaves as \emph{integral} control; our terms introduce \emph{proportional} and \emph{derivative} control, achieving favorable learning dynamics through damping and predictive measures. We apply our PID Lagrangian methods in deep RL, setting a new state of the art in Safety Gym, a safe RL benchmark. Lastly, we introduce a new method to ease controller tuning by providing invariance to the relative numerical scales of reward and cost. Our extensive experiments demonstrate improved performance and hyperparameter robustness, while our algorithms remain nearly as simple to derive and implement as the traditional Lagrangian approach.

Motivation & Objective

• Motivate the instability and oscillations observed with traditional Lagrangian updates in constrained RL.
• Propose a proportional-integral-derivative (PID) update for the Lagrange multiplier to dampen oscillations and improve responsiveness.
• Adapt the approach to deep RL and integrate with PPO to achieve safer, more robust training.
• Provide a reward-cost scale-invariance mechanism to ease hyperparameter tuning across environments.

Proposed method

• Model constrained RL as a dynamical system with the Lagrange multiplier as a control input.
• Introduce PID updates to the Lagrange multiplier by adding proportional and derivative terms to the standard integral update.
• Derive the modified dynamics showing the damping improvement and potential predictive benefits.
• Integrate the PID Lagrangian method with PPO to create Constraint-Controlled PPO (CPPO) in Safety Gym environments.
• Propose a reward-cost scale-invariance technique to make controller tuning robust to relative scales of reward and cost.

Experimental results

Research questions

• RQ1 Can PID-based updates to the Lagrange multiplier reduce cost overshoot and constraint violations during training?
• RQ2 Do PID-Lagrangian methods provide improved hyperparameter robustness and learning efficiency in safe RL benchmarks?
• RQ3 How does the approach perform when integrated with a leading deep RL algorithm (PPO) on Safety Gym tasks?
• RQ4 Can tuning be simplified via reward-cost scale invariance without sacrificing performance?

Key findings

• PID updates damp cost oscillations and overshoot compared to the traditional integral-only Lagrangian method.
• PI-control (nonzero P term) reduces constraint violations across multiple Safety Gym tasks while maintaining rewards.
• Derivative control offers predictive damping and can prevent cost overshoot, slowing the approach to the constraint boundary.
• Constraint-Controlled PPO with PID updates achieves state-of-the-art performance on Safety Gym benchmarks.
• A reward-scale invariance technique eases hyperparameter tuning and improves robustness across environments.