[Paper Review] Generalized discounted continuous-time Markov decision processes

Xianping Guo, Yi Zhang|arXiv (Cornell University)|Apr 11, 2013

Economic Policies and Impacts3 citations

TL;DR

This paper introduces a transformation method to solve generalized discounted continuous-time Markov decision processes (CTMDPs) in Polish spaces with unbounded rates and state-dependent or zero discount factors, enabling the analysis of total undiscounted criteria. It establishes existence of deterministic (unconstrained) and randomized (constrained) stationary optimal policies outside history-dependent classes, along with an optimality equation for the unconstrained case.

ABSTRACT

This article investigates the generalized discounted criteria for possibly explosive continuous-time Markov decision processes (CTMDPs) in Polish spaces with unbounded transition and reward rates, which allow the discount factors to be state-dependent and zero-valued, and thus cover the otherwise underdeveloped total undiscounted criteria for non-absorbing CTMDPs. Nontrivially, we, under very mild conditions, develop the transformation method, which reduces our CTMDP problems to their discrete-time analogues, and then show the existence of a deterministic (resp., randomized) stationary optimal policy for the unconstrained (resp., constrained) CTMDPs, where the optimality is out of the class of history-dependent policies. Moreover, the optimality equation for the unconstrained case is also established. Quite differently from the case of standard discounted CTMDPs with a constant discount factor, we show that the transformation method could be inapplicable to the concerned CTMDPs with generalized discount factors when our conditions are violated.

Motivation & Objective

• To address the lack of theoretical foundations for total undiscounted criteria in non-absorbing continuous-time Markov decision processes.
• To extend the theory of CTMDPs to allow unbounded transition and reward rates.
• To develop a transformation method that reduces continuous-time problems to discrete-time analogues under mild conditions.
• To establish existence of deterministic and randomized stationary optimal policies for unconstrained and constrained CTMDPs, respectively.
• To derive an optimality equation for the unconstrained generalized discounted CTMDP case.

Proposed method

• Proposes a transformation method that maps generalized discounted CTMDPs with state-dependent or zero discount factors into equivalent discrete-time MDPs.
• Applies the transformation under mild conditions to ensure the reduction is valid and preserves optimality.
• Uses dynamic programming principles to derive the optimality equation for the unconstrained case.
• Establishes existence of optimal policies within the class of stationary policies, excluding history-dependent ones.
• Analyzes the structural conditions under which the transformation method fails when assumptions are violated.
• Relies on measure-theoretic tools in Polish spaces to handle unbounded rates and general state spaces.

Experimental results

Research questions

• RQ1 Under what conditions can generalized discounted CTMDPs with state-dependent or zero discount factors be reduced to discrete-time MDPs via transformation?
• RQ2 Does a deterministic stationary optimal policy exist for unconstrained CTMDPs under unbounded transition and reward rates?
• RQ3 Can a randomized stationary optimal policy be guaranteed for constrained CTMDPs under the same conditions?
• RQ4 What is the form of the optimality equation for the unconstrained generalized discounted CTMDP?
• RQ5 Why does the transformation method fail when the mild conditions are violated?

Key findings

• A transformation method is developed that reduces generalized discounted CTMDPs with state-dependent or zero discount factors to discrete-time analogues under mild conditions.
• The existence of a deterministic stationary optimal policy is proven for unconstrained CTMDPs with unbounded rates.
• A randomized stationary optimal policy is established for constrained CTMDPs under the same framework.
• An optimality equation is derived for the unconstrained case, extending classical results to generalized discounting.
• The transformation method is shown to be inapplicable when the mild conditions are violated, highlighting the necessity of these assumptions.
• The results generalize standard discounted CTMDPs with constant discount factors to broader, previously underdeveloped cases.