[Paper Review] Finite Optimal Control for Time-Bounded Reachability in CTMDPs and Continuous-Time Markov Games

Markus N. Rabe, Sven Schewe|arXiv (Cornell University)|Apr 22, 2010

Reinforcement Learning in Robotics11 references4 citations

TL;DR

This paper establishes the existence of finite optimal control strategies for time-bounded reachability in continuous-time Markov decision processes (CTMDPs) and continuous-time Markov games (CTMGs). It proves that optimal schedulers are deterministic, timed-positional, and piecewise positional over a finite partition of the time horizon, ensuring finite optimal control even in the presence of opposing objectives in games.

ABSTRACT

We establish the existence of optimal scheduling strategies for time-bounded reachability in continuous-time Markov decision processes, and of co-optimal strategies for continuous-time Markov games. Furthermore, we show that optimal control does not only exist, but has a surprisingly simple structure: The optimal schedulers from our proofs are deterministic and timed-positional, and the bounded time can be divided into a finite number of intervals, in which the optimal strategies are positional. That is, we demonstrate the existence of finite optimal control. Finally, we show that these pleasant properties of Markov decision processes extend to the more general class of continuous-time Markov games, and that both early and late schedulers show this behaviour.

Motivation & Objective

• To resolve the long-standing open question of whether optimal control exists for time-bounded reachability in CTMDPs.
• To extend the existence of optimal control to the more complex setting of continuous-time Markov games (CTMGs) with opposing players.
• To demonstrate that optimal strategies are not only existent but possess a finite, structured form—specifically, deterministic and timed-positional with finitely many switching intervals.
• To unify and generalize existing results by showing that both early and late scheduler models yield the same optimal control structure.
• To provide a topological and compactness-based proof framework that supports finiteness and measurability of optimal strategies.

Proposed method

• Introduces discrete locations in CTMDPs to simplify scheduler modeling and enable translation between early and late schedulers.
• Uses topological arguments to prove existence of measurable optimal schedulers by fixing decisions on closures of open sets.
• Applies compactness of the bounded time interval to show that optimal strategies are piecewise positional over a finite number of time intervals.
• Derives backward Kolmogorov-type differential equations for optimal value functions: $-\dot{f}_{\mathsf{opt}}(l,t) = \max_{a} \sum_{l'} \mathbf{R}(l,a,l') \cdot (f_{\mathsf{opt}}(l',t) - f_{\mathsf{opt}}(l,t))$ for reachability and $\min$ for safety.
• Lifts results to CTMGs by showing that Nash equilibria must satisfy the same differential equations, and proves existence of cylindrical, deterministic, timed-positional co-optimal strategies.

Experimental results

Research questions

• RQ1 Do optimal control strategies exist for time-bounded reachability in continuous-time Markov decision processes (CTMDPs) with general schedulers?
• RQ2 Can the structure of optimal strategies in CTMDPs be characterized as finite and positional over a partition of the time horizon?
• RQ3 Does the existence and finiteness of optimal control extend to continuous-time Markov games (CTMGs) with opposing objectives?
• RQ4 Can the same structural properties—deterministic, timed-positional, finite switching points—be preserved in CTMGs under both early and late scheduler semantics?
• RQ5 What is the impact of infinite states or actions on the finiteness of optimal strategies in time-bounded reachability problems?

Key findings

• Optimal schedulers for time-bounded reachability in CTMDPs exist and are deterministic, timed-positional, and piecewise positional over a finite partition of the time interval.
• The optimal control strategy can be computed via backward solution of a system of differential equations: $-\dot{f}_{\mathsf{opt}}(l,t) = \max_{a} \sum_{l'} \mathbf{R}(l,a,l') \cdot (f_{\mathsf{opt}}(l',t) - f_{\mathsf{opt}}(l,t))$ for reachability.
• For CTMGs, co-optimal strategies exist for both players and are cylindrical, deterministic, and timed-positional, forming a Nash equilibrium.
• The differential equations governing optimal value functions are invariant under local uniformization of rates, allowing for computational simplification.
• In the presence of infinitely many states or actions, optimal strategies may require infinitely many switching points, showing the necessity of finite state/action assumptions for finiteness.
• The results extend to non-absorbing goal regions by modifying the terminal condition to $f_{\mathsf{opt}}(l,t_{\max}) = 1$ for goal locations, while preserving the same differential equations.