[Paper Review] Computing Quantiles in Markov Reward Models

Michael Ummels, Christel Baier|arXiv (Cornell University)|Jan 9, 2013

Formal Methods in Verification1 references6 citations

TL;DR

This paper introduces quantile queries for Markov reward models, where the goal is to compute the minimal reward bound r such that a target state is reached with probability at least p before r is exceeded. It presents a polynomial-time algorithm for qualitative quantile queries and a pseudo-polynomial-time algorithm for quantitative queries in Markov chains, with exponential-time solutions for Markov decision processes.

ABSTRACT

Probabilistic model checking mainly concentrates on techniques for reasoning about the probabilities of certain path properties or expected values of certain random variables. For the quantitative system analysis, however, there is also another type of interesting performance measure, namely quantiles. A typical quantile query takes as input a lower probability bound p and a reachability property. The task is then to compute the minimal reward bound r such that with probability at least p the target set will be reached before the accumulated reward exceeds r. Quantiles are well-known from mathematical statistics, but to the best of our knowledge they have not been addressed by the model checking community so far. In this paper, we study the complexity of quantile queries for until properties in discrete-time finite-state Markov decision processes with non-negative rewards on states. We show that qualitative quantile queries can be evaluated in polynomial time and present an exponential algorithm for the evaluation of quantitative quantile queries. For the special case of Markov chains, we show that quantitative quantile queries can be evaluated in time polynomial in the size of the chain and the maximum reward.

Motivation & Objective

• To address the lack of direct support for quantile-based queries in probabilistic model checking, which are crucial for performance and reliability analysis in systems.
• To formalize and analyze quantile queries of the form P_≥p(a U_≤? b), where p is a probability threshold and ? denotes the minimal reward bound.
• To determine the computational complexity of such queries in discrete-time Markov chains and Markov decision processes with nonnegative rewards.
• To provide efficient algorithms for evaluating these queries, especially in the context of real-time and resource-constrained system analysis.
• To extend existing PCTL and PRCTL frameworks by introducing quantile queries as a new primitive for quantitative system verification.

Proposed method

• Formalize quantile queries as the minimal reward bound r such that the probability of reaching a target state via a path with accumulated reward ≤ r is at least p.
• Use dynamic programming along the binary representation of r to compute reachability probabilities Pr_s(a U_=r b) in Markov chains with rewards 0 and 1.
• Transform general Markov chains with rewards in {0,1,...,c} into equivalent chains with rewards 0 and 1, preserving reachability probabilities.
• Apply binary search over the reward bound r to compute the minimal r satisfying the probability constraint, leveraging bounds on the precision of p and q.
• Establish theoretical bounds on the size of the reward bound r using logarithmic terms in the probability difference (q−p), enabling efficient search.
• Adapt the algorithm to handle both strict and non-strict probability bounds (e.g., >p and ≥p) via case analysis and known probability bounds from Lemma 5.

Experimental results

Research questions

• RQ1 What is the computational complexity of evaluating quantile queries P_≥p(a U_≤? b) in discrete-time Markov chains with nonnegative rewards?
• RQ2 Can qualitative quantile queries (where p is 0 or 1) be evaluated in polynomial time in Markov decision processes?
• RQ3 What is the complexity of quantitative quantile queries in Markov decision processes, and can they be solved more efficiently than brute-force search?
• RQ4 How can the minimal reward bound r be computed efficiently such that the probability of reaching a target state within reward r is at least p?
• RQ5 Are there theoretical bounds on the size of the minimal reward bound r that can be used to guide a search-based algorithm?

Key findings

• Qualitative quantile queries (p = 0 or 1) in Markov decision processes can be evaluated in polynomial time using graph-theoretic analysis.
• For Markov chains, quantitative quantile queries can be computed in pseudo-polynomial time, specifically O(poly(c·|M|·‖p‖)), where c is the maximum reward and ‖p‖ is the bit-length of p.
• The algorithm uses binary search over the reward bound r, with each step requiring O(poly(c·|M|)·log r) time to compute reachability probabilities.
• The search space is bounded by r ≤ k·n·c, where k = max{−⌊ln(q−p)⌋, 1}, n is the number of a-labeled states, and c is the maximum reward, ensuring logarithmic search depth.
• The method generalizes to both strict and non-strict probability bounds (e.g., >p and ≥p) by handling edge cases via Lemma 5.
• The results extend to dual queries involving existential quantification over schedulers, via duality with reward-bounded until properties.