[Paper Review] Quantile Markov Decision Process

Xiaocheng Li, Huaiyang Zhong+1 more|arXiv (Cornell University)|Nov 15, 2017

Advanced Causal Inference Techniques48 references3 citations

TL;DR

This paper introduces the Quantile Markov Decision Process (QMDP), a novel framework that optimizes specific quantiles of cumulative rewards in MDPs rather than the traditional expected value. By augmenting the state space to track past cumulative rewards, the authors develop a dynamic programming algorithm that efficiently computes optimal policies for varying risk attitudes, with applications in personalized HIV treatment planning showing how risk preferences shape treatment timing.

ABSTRACT

The goal of a traditional Markov decision process (MDP) is to maximize expected cumulative reward over a defined horizon (possibly infinite). In many applications, however, a decision maker may be interested in optimizing a specific quantile of the cumulative reward instead of its expectation. In this paper we consider the problem of optimizing the quantiles of the cumulative rewards of a Markov decision process (MDP), which we refer to as a quantile Markov decision process (QMDP). We provide analytical results characterizing the optimal QMDP value function and present a dynamic programming-based algorithm to solve for the optimal policy. The algorithm also extends to the MDP problem with a conditional value-at-risk (CVaR) objective. We illustrate the practical relevance of our model by evaluating it on an HIV treatment initiation problem, where patients aim to balance the potential benefits and risks of the treatment.

Motivation & Objective

• To address the limitation of traditional MDPs, which optimize expected cumulative rewards, by introducing a framework that instead maximizes specific quantiles of the reward distribution.
• To model risk-sensitive decision-making by incorporating patient or system-level risk preferences into sequential decision problems.
• To develop an efficient dynamic programming algorithm that solves for optimal policies across all quantiles in a single pass, leveraging state augmentation.
• To extend the framework to conditional value-at-risk (CVaR) objectives, enabling robustness to tail risks.
• To demonstrate practical relevance through an HIV treatment initiation case study, showing how risk aversion influences treatment timing.

Proposed method

• The QMDP model augments the state space to include a performance measure of past cumulative rewards, transforming the non-Markovian quantile objective into a Markovian one.
• The algorithm uses dynamic programming to compute the optimal value function for all states and quantiles simultaneously, with time complexity O(AST·max(RT, S)).
• The augmented state tracks the cumulative reward up to each stage, enabling the policy to adapt as 'aggressive', 'neutral', or 'conservative' based on risk preferences.
• The method extends naturally to CVaR objectives by leveraging nested risk measures and recursive value updates.
• The approach is validated through simulation on an HIV treatment model with 50,000 reward trajectories to compare MDP and QMDP outcomes.
• Policy stability is analyzed, with observed instabilities near action switches attributed to reward and transition structure.

Experimental results

Research questions

• RQ1 How can a Markov decision process be reformulated to optimize a specific quantile of cumulative rewards rather than the expected value?
• RQ2 What state augmentation strategy enables dynamic programming to solve the quantile optimization problem efficiently?
• RQ3 How does the optimal policy derived from QMDP reflect varying levels of risk aversion in sequential decision-making?
• RQ4 What is the relationship between the MDP’s expected reward distribution and the QMDP solution across different quantiles?
• RQ5 Can the QMDP framework be extended to handle conditional value-at-risk (CVaR) objectives in risk-averse decision-making?

Key findings

• The QMDP model successfully computes optimal policies for all quantiles in a single dynamic programming pass by augmenting the state space with cumulative reward history.
• For a 60-year-old woman with CD4 count 200, a risk-averse patient (τ=0.20) delays ART initiation, while a less risk-averse patient (τ=0.80) starts earlier, demonstrating risk preference-driven policy divergence.
• The CDF of MDP rewards shows greater potential for improvement at higher quantiles, indicating that QMDP can yield substantial gains when decision-makers are less risk-averse.
• The optimal policy exhibits instability near action switches, likely due to the structure of immediate and terminal rewards and transition probabilities, suggesting need for further conditions on policy monotonicity.
• The QMDP framework provides a practical tool for comparing MDP and risk-sensitive solutions, revealing how much improvement could be gained by using quantile objectives instead of expectation.
• The method extends to CVaR objectives, enabling robust optimization under tail-risk constraints, with potential for integration with approximate dynamic programming in large-scale problems.