[Paper Review] Policy Gradients Beyond Expectations: Conditional Value-at-Risk.

Aviv Tamar, Yonatan Glassner+1 more|arXiv (Cornell University)|Apr 15, 2014

Risk and Portfolio OptimizationDecision Sciences26 references20 citations

TL;DR

This paper introduces a novel policy gradient method for Conditional Value-at-Risk (CVaR) optimization by deriving a gradient formula as a conditional expectation, enabling sampling-based estimation and gradient descent for risk-sensitive reinforcement learning. The approach is validated in a Tetris controller learning task with an importance sampling procedure for improved sample efficiency.

ABSTRACT

Conditional Value at Risk (CVaR) is a prominent risk measure that is being used extensively in various domains such as finance. In this work we present a new formula for the gradient of the CVaR in the form of a conditional expectation. Our result is similar to policy gradients in the reinforcement learning literature. Based on this formula, we propose novel sampling-based estimators for the CVaR gradient, and a corresponding gradient descent procedure for CVaR optimization. We evaluate our approach in learning a risk-sensitive controller for the game of Tetris, and propose an importance sampling procedure that is suitable for such domains.

Motivation & Objective

• To develop a principled gradient estimation method for CVaR in sequential decision-making under uncertainty.
• To extend policy gradient reinforcement learning to risk-sensitive objectives using CVaR as a measure of tail risk.
• To enable practical optimization of CVaR through sampling-based estimators and a gradient descent procedure.
• To improve sample efficiency in risk-sensitive control via an importance sampling procedure tailored for CVaR objectives.

Proposed method

• Derives a new analytical formula for the gradient of CVaR as a conditional expectation, analogous to standard policy gradients.
• Proposes sampling-based estimators that leverage the derived gradient formula to approximate the true gradient in practice.
• Designs a gradient descent algorithm for optimizing CVaR objectives in reinforcement learning settings.
• Introduces an importance sampling procedure to improve sample efficiency when estimating CVaR gradients in simulation environments.
• Applies the method to train a risk-sensitive controller in the Tetris environment, demonstrating robustness to high-variance outcomes.
• Uses the conditional expectation structure to enable stable and scalable optimization of risk-averse policies.

Experimental results

Research questions

• RQ1 How can the gradient of the CVaR objective be expressed in a form suitable for policy gradient reinforcement learning?
• RQ2 What sampling-based estimators can be derived from the new gradient formula to enable practical optimization?
• RQ3 How can importance sampling be adapted to improve sample efficiency in CVaR-based reinforcement learning?
• RQ4 Can the proposed method effectively learn risk-sensitive policies in environments with high outcome variance, such as Tetris?
• RQ5 What performance gains does the method achieve in terms of risk mitigation compared to standard policy gradient approaches?

Key findings

• The paper successfully derives a closed-form gradient expression for CVaR as a conditional expectation, enabling gradient-based optimization.
• The proposed sampling-based estimators provide a stable and effective way to approximate the CVaR gradient in practice.
• The gradient descent procedure based on the new formula enables optimization of risk-averse policies in reinforcement learning.
• The importance sampling procedure improves sample efficiency in estimating CVaR gradients, particularly in high-variance environments.
• The method achieves improved risk sensitivity in learning a Tetris controller, demonstrating robustness to rare but costly failure outcomes.
• Empirical results show that the approach effectively reduces tail risk compared to standard policy gradient methods, as measured by CVaR performance.