[Paper Review] Risk-Constrained Reinforcement Learning with Percentile Risk Criteria

Yinlam Chow, Mohammad Ghavamzadeh+2 more|arXiv (Cornell University)|Dec 5, 2015

Reinforcement Learning in Robotics40 references54 citations

TL;DR

This paper proposes policy gradient and actor-critic algorithms for risk-constrained reinforcement learning using percentile risk criteria, specifically chance constraints and conditional value-at-risk (CVaR). It derives gradient estimators for the Lagrangian, enables joint policy and multiplier updates, and proves convergence to locally optimal policies in risk-constrained Markov decision processes.

ABSTRACT

In many sequential decision-making problems one is interested in minimizing an expected cumulative cost while taking into account \emph{risk}, i.e., increased awareness of events of small probability and high consequences. Accordingly, the objective of this paper is to present efficient reinforcement learning algorithms for risk-constrained Markov decision processes (MDPs), where risk is represented via a chance constraint or a constraint on the conditional value-at-risk (CVaR) of the cumulative cost. We collectively refer to such problems as percentile risk-constrained MDPs. Specifically, we first derive a formula for computing the gradient of the Lagrangian function for percentile risk-constrained MDPs. Then, we devise policy gradient and actor-critic algorithms that (1) estimate such gradient, (2) update the policy in the descent direction, and (3) update the Lagrange multiplier in the ascent direction. For these algorithms we prove convergence to locally optimal policies. Finally, we demonstrate the effectiveness of our algorithms in an optimal stopping problem and an online marketing application.

Motivation & Objective

• Address the gap in reinforcement learning for risk-constrained Markov decision processes (MDPs), where risk is defined via chance constraints or CVaR.
• Develop efficient, scalable RL algorithms that handle percentile risk criteria while maintaining computational tractability.
• Enable joint optimization of policies and Lagrange multipliers through gradient-based methods in risk-constrained settings.
• Provide theoretical convergence guarantees for the proposed algorithms under standard stochastic approximation assumptions.
• Demonstrate effectiveness on real-world sequential decision-making problems involving rare but high-impact events.

Proposed method

• Formulate risk-constrained MDPs using chance constraints and CVaR as risk metrics, embedding risk awareness into the objective function.
• Derive the gradient of the Lagrangian function for percentile risk-constrained MDPs, enabling gradient-based policy optimization.
• Design a policy gradient algorithm that estimates the gradient of the Lagrangian and updates the policy in the negative gradient direction.
• Develop an actor-critic algorithm that combines value function approximation with policy gradient updates for improved sample efficiency.
• Implement a three-time-scale stochastic approximation scheme: fast for policy (θ), medium for value function (v), slowest for Lagrange multiplier (λ).
• Use the γ-occupation measure to generate unbiased gradient estimates and ensure convergence via martingale difference error terms.

Experimental results

Research questions

• RQ1 How can risk-constrained MDPs with percentile risk criteria be formulated and solved efficiently using reinforcement learning?
• RQ2 What is the correct gradient of the Lagrangian for risk-constrained MDPs involving CVaR and chance constraints?
• RQ3 Can policy gradient and actor-critic algorithms be adapted to jointly optimize policies and Lagrange multipliers in risk-constrained settings?
• RQ4 What convergence guarantees can be established for such algorithms under stochastic approximation?
• RQ5 How do the proposed algorithms perform in practical applications involving rare but costly events?

Key findings

• The proposed policy gradient and actor-critic algorithms converge almost surely to locally optimal policies under standard stochastic approximation conditions.
• The gradient of the Lagrangian for percentile risk-constrained MDPs is derived and used to enable joint policy and multiplier updates.
• The three-time-scale update scheme ensures that policy, value function, and Lagrange multiplier updates converge independently, with the multiplier updating on the slowest time scale.
• Empirical results show the algorithms outperform risk-neutral baselines in an optimal stopping problem and an online marketing application, particularly in reducing tail risk.
• The method effectively enforces CVaR and chance constraints, ensuring that high-cost events are minimized even when they occur with low probability.
• Theoretical analysis confirms that the error terms in the updates are martingale differences with vanishing bias, supporting convergence to a local saddle point.