[Paper Review] Energy and Mean-Payoff Parity Markov Decision Processes

Krishnendu Chatterjee, Laurent Doyen|arXiv (Cornell University)|Jan 1, 2011

Formal Methods in Verification16 references10 citations

TL;DR

This paper studies Markov decision processes (MDPs) with combined parity, energy, and mean-payoff objectives to model reactive systems with quantitative resource constraints and ω-regular specifications. It shows that deciding almost-sure winning states for mean-payoff parity MDPs is solvable in polynomial time, while for energy parity MDPs the problem lies in NP ∩ coNP, with a pseudo-polynomial algorithm, resolving complexity gaps and improving prior PSPACE bounds.

ABSTRACT

We consider Markov Decision Processes (MDPs) with mean-payoff parity and energy parity objectives. In system design, the parity objective is used to encode ω-regular specifications, and the mean-payoff and energy objectives can be used to model quantitative resource constraints. The energy condition requires that the resource level never drops below 0, and the mean-payoff condition requires that the limit-average value of the resource consumption is within a threshold. While these two (energy and mean-payoff) classical conditions are equivalent for two-player games, we show that they differ for MDPs. We show that the problem of deciding whether a state is almost-sure winning (i.e., winning with probability 1) in energy parity MDPs is in NP \cap coNP, while for mean-payoff parity MDPs, the problem is solvable in polynomial time, improving a recent PSPACE bound.

Motivation & Objective

• To analyze the computational complexity of deciding almost-sure winning strategies in MDPs with combined parity and quantitative objectives (energy or mean-payoff).
• To resolve the complexity gap for energy parity MDPs, showing the problem is in NP ∩ coNP and solvable in pseudo-polynomial time.
• To improve the PSPACE upper bound from prior work by providing a polynomial-time algorithm for mean-payoff parity MDPs.
• To characterize the memory requirements of almost-sure winning strategies for both energy and mean-payoff parity objectives.
• To establish tight complexity bounds by relating the problems to parity games and two-player energy games.

Proposed method

• Uses end-component analysis to decompose the MDP into strongly connected sub-MDPs that support almost-sure winning strategies.
• Iteratively computes winning end-components by prioritizing even priorities, starting from the smallest even priority.
• For mean-payoff parity MDPs, computes winning end-components by checking whether the maximal expected mean-payoff value in each end-component is at least ν.
• Employs linear programming to compute mean-payoff values in sub-MDPs, enabling polynomial-time computation.
• Reduces energy parity MDPs to two-player energy B¨uchi games, leveraging known NP ∩ coNP algorithms.
• Applies attractor and sub-MDP analysis to compute almost-sure reachability to winning end-components.

Experimental results

Research questions

• RQ1 What is the computational complexity of deciding almost-sure winning states in energy parity MDPs?
• RQ2 Can the problem of deciding almost-sure winning for mean-payoff parity MDPs be solved in polynomial time, improving upon prior PSPACE bounds?
• RQ3 What is the memory requirement for almost-sure winning strategies in energy and mean-payoff parity MDPs?
• RQ4 How do energy and mean-payoff objectives differ in MDPs, despite being equivalent in two-player games?
• RQ5 Can winning end-components be computed efficiently to enable polynomial-time solution for mean-payoff parity objectives?

Key findings

• The problem of deciding almost-sure winning states for mean-payoff parity MDPs is solvable in polynomial time, providing a tight PTIME-complete upper bound.
• For energy parity MDPs, the problem is in NP ∩ coNP, and a pseudo-polynomial time algorithm is presented, matching the best-known upper bounds unless parity games are in P.
• Almost-sure winning strategies for mean-payoff parity objectives with non-strict inequality (≥ν) require infinite memory in general, while finite-memory strategies suffice for strict inequality (>ν).
• Winning end-components are defined as those with even minimal priority and sub-MDPs where the maximal expected mean-payoff value is at least ν.
• The value function for mean-payoff parity MDPs can be computed by reducing it to almost-sure reachability to the union of winning end-components, which is computable in polynomial time.
• The disjunction of energy and parity objectives is decidable in NP ∩ coNP, and the disjunction of mean-payoff and parity objectives is solvable in polynomial time.