[Paper Review] Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach

Yinlam Chow, Aviv Tamar+2 more|arXiv (Cornell University)|Jun 6, 2015

Risk and Portfolio Optimization25 references101 citations

TL;DR

This paper proposes a novel CVaR optimization framework for Markov decision processes (MDPs) that unifies risk-sensitive and robust decision-making by interpreting CVaR as minimizing expected cost under worst-case model perturbations within a bounded error budget. It introduces an approximate value-iteration algorithm with provable convergence and finite-time error bounds, enabling globally optimal policy computation for non-trivial MDPs with thousands of states.

ABSTRACT

In this paper we address the problem of decision making within a Markov decision process (MDP) framework where risk and modeling errors are taken into account. Our approach is to minimize a risk-sensitive conditional-value-at-risk (CVaR) objective, as opposed to a standard risk-neutral expectation. We refer to such problem as CVaR MDP. Our first contribution is to show that a CVaR objective, besides capturing risk sensitivity, has an alternative interpretation as expected cost under worst-case modeling errors, for a given error budget. This result, which is of independent interest, motivates CVaR MDPs as a unifying framework for risk-sensitive and robust decision making. Our second contribution is to present an approximate value-iteration algorithm for CVaR MDPs and analyze its convergence rate. To our knowledge, this is the first solution algorithm for CVaR MDPs that enjoys error guarantees. Finally, we present results from numerical experiments that corroborate our theoretical findings and show the practicality of our approach.

Motivation & Objective

• To unify risk-sensitive and robust decision-making in MDPs by showing that CVaR minimization is equivalent to worst-case expected cost under bounded model perturbations.
• To develop a computationally tractable algorithm for solving CVaR MDPs that provides finite-time error guarantees.
• To overcome the challenges of continuous augmented state spaces in prior CVaR MDP methods through interpolation-based approximate value iteration.
• To demonstrate the practicality and scalability of the approach on large-scale grid-world planning problems.
• To extend the framework to mean-CVaR objectives and provide a Bellman recursion with convergence guarantees.

Proposed method

• Formulates a CVaR MDP using state augmentation with a continuous auxiliary variable representing the confidence level of the risk measure.
• Derives a Bellman equation for CVaR MDPs based on a risk envelope that captures worst-case perturbations of transition probabilities within a bounded error budget.
• Proposes an approximate value-iteration algorithm that combines linear interpolation over the augmented state space to handle continuous variables.
• Establishes contraction-style convergence analysis with explicit error bounds, proving convergence to the globally optimal policy within finite time.
• Extends the framework to mean-CVaR objectives by introducing a two-dimensional risk envelope and a composite risk operator.
• Uses a linear operator Sλ to combine multiple CVaR confidence levels, enabling joint optimization over mean and tail risk.

Experimental results

Research questions

• RQ1 Can CVaR minimization in MDPs be interpreted as robust optimization under worst-case model perturbations?
• RQ2 Can a value-iteration algorithm for CVaR MDPs be designed with provable finite-time error bounds despite continuous state augmentation?
• RQ3 How does the proposed interpolation-based algorithm compare to prior methods in terms of computational complexity and convergence speed?
• RQ4 Can the framework be extended to multi-objective risk measures such as mean-CVaR with explicit convergence guarantees?
• RQ5 What is the empirical performance of the algorithm on large-scale planning problems with thousands of states?

Key findings

• The CVaR of the discounted cost in an MDP is mathematically equivalent to the expected cost under worst-case perturbations of the transition probabilities, provided the perturbations are bounded by a given error budget.
• The proposed approximate value-iteration algorithm converges to the globally optimal policy with a finite-time error bound of the form $ \frac{\gamma^n}{1-\gamma}(C_{\text{max}} + \|Z\|_\infty) $, where $ \gamma $ is the discount factor and $ Z $ is the initial cost random variable.
• The algorithm achieves convergence with explicit error guarantees, unlike prior methods that only converge asymptotically or require solving non-convex programs.
• Numerical experiments on a grid-world with thousands of states confirm the algorithm’s practicality and scalability, demonstrating its ability to compute globally optimal policies.
• The extension to mean-CVaR objectives is supported by a new Bellman recursion with a composite risk envelope, and the fixed point solution is proven to be unique and optimal.
• The algorithm computes optimal policies for all CVaR confidence levels and initial states simultaneously, significantly improving efficiency over per-confidence-level computation.