[Paper Review] Algorithms for CVaR Optimization in MDPs

Yinlam Chow, Mohammad Ghavamzadeh|arXiv (Cornell University)|Jun 12, 2014

Risk and Portfolio OptimizationDecision Sciences30 references67 citations

TL;DR

This paper proposes policy gradient and actor-critic algorithms for mean-CVaR optimization in Markov decision processes (MDPs), deriving a novel gradient formula for the risk-sensitive objective. It establishes convergence to locally optimal policies and demonstrates effectiveness in an optimal stopping problem, offering incremental and trajectory-based estimation with rigorous theoretical guarantees—extending prior work by providing convergence proofs and broader applicability to both discrete and continuous loss distributions.

ABSTRACT

In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in costs in addition to minimizing a standard criterion. Conditional value-at-risk (CVaR) is a relatively new risk measure that addresses some of the shortcomings of the well-known variance-related risk measures, and because of its computational efficiencies has gained popularity in finance and operations research. In this paper, we consider the mean-CVaR optimization problem in MDPs. We first derive a formula for computing the gradient of this risk-sensitive objective function. We then devise policy gradient and actor-critic algorithms that each uses a specific method to estimate this gradient and updates the policy parameters in the descent direction. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in an optimal stopping problem.

Motivation & Objective

• To address risk-sensitive sequential decision-making by optimizing a mean-CVaR objective in MDPs, which balances expected cost and tail risk.
• To derive a closed-form gradient of the mean-CVaR objective function for policy parameter updates.
• To develop both policy gradient and actor-critic algorithms that estimate the gradient incrementally or via trajectories.
• To establish asymptotic convergence of the proposed algorithms to locally risk-sensitive optimal policies.
• To demonstrate the practical utility of the algorithms in a non-trivial optimal stopping problem.

Proposed method

• Derives the gradient of the mean-CVaR objective using the policy’s value function and advantage function, enabling policy parameter updates.
• Uses the occupation measure and expected return formulation to express the gradient in terms of state-action visitation frequencies.
• Proposes both incremental and trajectory-based gradient estimators for use in policy gradient and actor-critic frameworks.
• Applies the ordinary differential equations (ODE) approach to prove asymptotic convergence of the algorithms.
• Introduces a projected equation formulation for value function approximation using function approximation (e.g., linear function approximation with basis functions).
• Employs the advantage function decomposition to express the gradient in terms of policy log-derivative and action-value function.

Experimental results

Research questions

• RQ1 How can the gradient of the mean-CVaR objective in MDPs be analytically derived for policy optimization?
• RQ2 Can policy gradient and actor-critic algorithms be designed to optimize mean-CVaR with convergence guarantees?
• RQ3 How do incremental and trajectory-based gradient estimation methods compare in performance and variance reduction?
• RQ4 What is the theoretical convergence behavior of the proposed algorithms under stochastic approximation?
• RQ5 How does the proposed method perform in practical risk-sensitive control tasks like optimal stopping?

Key findings

• The paper derives a closed-form gradient expression for the mean-CVaR objective, enabling policy parameter updates via policy gradient methods.
• The proposed policy gradient and actor-critic algorithms converge asymptotically to locally risk-sensitive optimal policies under the ODE approach.
• The algorithms support both discrete and continuous loss distributions, extending prior work limited to continuous cases.
• The actor-critic variant reduces variance compared to standard policy gradient, improving sample efficiency.
• The trajectory-based and incremental estimation methods are both effective, with the latter enabling online learning.
• Empirical results in an optimal stopping problem confirm the practical utility and robustness of the proposed algorithms.