[Paper Review] Reward Constrained Policy Optimization

Chen Tessler, Daniel J. Mankowitz+1 more|arXiv (Cornell University)|May 28, 2018

Reinforcement Learning in Robotics29 references235 citations

TL;DR

RCPO introduces a multi-timescale constrained policy optimization that uses a discounted penalty signal to steer policies toward feasibility, with theoretical convergence guarantees and improved empirical performance over standard methods.

ABSTRACT

Solving tasks in Reinforcement Learning is no easy feat. As the goal of the agent is to maximize the accumulated reward, it often learns to exploit loopholes and misspecifications in the reward signal resulting in unwanted behavior. While constraints may solve this issue, there is no closed form solution for general constraints. In this work we present a novel multi-timescale approach for constrained policy optimization, called `Reward Constrained Policy Optimization' (RCPO), which uses an alternative penalty signal to guide the policy towards a constraint satisfying one. We prove the convergence of our approach and provide empirical evidence of its ability to train constraint satisfying policies.

Motivation & Objective

• Motivate constrained reinforcement learning to prevent undesirable reward exploitation and ensure constraint satisfaction.
• Propose a penalty-based approach that integrates constraints into the reward signal without requiring prior tuning of penalty coefficients.
• Establish convergence guarantees for RCPO under mild assumptions.
• Demonstrate empirical benefits of RCPO across grid-world and Mujoco robotics domains.
• Compare RCPO to reward shaping and standard constrained optimization baselines.

Proposed method

• Formulate constrained MDPs (CMDP) and apply a Lagrangian relaxation to create a two-timescale update: fast policy parameters θ and slow penalty λ.
• Introduce a discounted guiding penalty through Cγ and define penalized rewards r̂(λ, s, a) = r(s,a) − λ c(s,a).
• Estimate gradients via policy gradient with log-likelihood tricks and a TD-based critic using the penalized value V̂π(λ, s).
• Operate a three-timescale RCPO: fast actor-critic updates, intermediate policy optimization, and slow λ updates to satisfy JＣπ ≤ α.
• Prove almost-sure convergence to a feasible local minimum (Theorem 2) under assumptions relating the guiding penalty to the actual constraint.
• Provide an RCPO algorithm template (Algorithm 1) and an RCPO Advantage Actor-Critic variant (Appendix A).

Experimental results

Research questions

• RQ1 Can a penalty-based, reward-guided approach satisfy generalized constraints in CMDPs without manually tuning penalties?
• RQ2 Does RCPO converge to constraint-feasible policies in practice and with what sample efficiency?
• RQ3 How does RCPO perform compared to reward shaping and traditional constraint methods across discrete (grid-world) and continuous (Mujoco) domains?
• RQ4 What are the effects of using a discounted guiding penalty on stability and convergence in constrained RL?
• RQ5 Can RCPO handle both discounted-sum and mean-value constraints in practice?

Key findings

• RCPO converges almost surely to a constraint-feasible fixed point under mild assumptions (Theorem 2).
• In experiments, RCPO achieves constraint-satisfying policies with faster convergence and lower variance than standard constrained optimization baselines in a grid-world Mars Rover domain.
• In Mujoco robotics tasks, RCPO finds feasible (or near-feasible) torque-constrained policies and generally outperforms fixed-λ reward shaping approaches across several environments (Swimmer, Walker2d, Hopper, Humanoid, HalfCheetah, Ant).
• Constant penalty methods (reward shaping) require domain-specific tuning and may lead to suboptimal or unstable policies as training progresses, whereas RCPO adapts λ to balance reward and constraint satisfaction.
• RCPO demonstrates improved sample efficiency and stability; it avoids the need for manual penalty coefficient tuning and handles general constraints beyond those satisfying recursive Bellman equations.