[Paper Review] Parameter Synthesis for Markov Models: Faster Than Ever

Tim Quatmann, Christian Dehnert+3 more|arXiv (Cornell University)|Feb 16, 2016

Formal Methods in Verification16 references6 citations

TL;DR

This paper introduces parameter lifting, a novel technique for parameter synthesis in parametric Markov chains and Markov decision processes (MDPs) that replaces parametric transitions with nondeterministic choices over extremal parameter values. By analyzing the resulting parameter-free models using off-the-shelf verification tools, the method efficiently computes tight lower and upper bounds on reachability probabilities across the parameter space, achieving orders-of-magnitude speedup and scalability to millions of states and complex parameter dependencies—providing the first sound and feasible approach for parametric MDPs.

ABSTRACT

We propose a simple technique for verifying probabilistic models whose transition probabilities are parametric. The key is to replace parametric transitions by nondeterministic choices of extremal values. Analysing the resulting parameter-free model using off-the-shelf means yields (refinable) lower and upper bounds on probabilities of regions in the parameter space. The technique outperforms the existing analysis of parametric Markov chains by several orders of magnitude regarding both run-time and scalability. Its beauty is its applicability to various probabilistic models. It in particular provides the first sound and feasible method for performing parameter synthesis of Markov decision processes.

Motivation & Objective

• To address the critical scalability limitations of existing parametric model checking techniques, which struggle with large models and complex parameter dependencies.
• To provide a sound and efficient method for parameter synthesis in parametric Markov decision processes (MDPs), which had previously been infeasible due to unsound heuristics in prior approaches.
• To enable high-coverage analysis of the parameter space (≥95%) for probabilistic systems with multi-affine polynomial parameters, particularly in models with dependent parameters.
• To overcome the computational infeasibility of rational function computation and SMT solving in state-of-the-art tools by replacing symbolic manipulation with numerical verification of relaxed models.
• To demonstrate that numerical verification of nondeterministic relaxations can yield precise, refinable bounds on parameter space regions without sacrificing correctness.

Proposed method

• Replace each parametric transition in a Markov model with a nondeterministic choice between the minimum and maximum values of the parameter function over its domain, creating a relaxed, parameter-free model.
• Construct a parameter-free MDP (or two-player stochastic game for MDPs) by equipping each state with a fresh parameter and replacing probabilistic choices with extremal nondeterministic choices.
• Apply standard numerical algorithms—such as value iteration or policy iteration—on the relaxed model to compute minimal and maximal reachability probabilities over the parameter space.
• Use the resulting bounds to classify regions of the parameter space as safe, unsafe, or undecided, with refinements via region splitting to improve coverage.
• Leverage the structure of multi-affine polynomials to ensure correctness and efficiency, as these functions are common in practical benchmarks.
• Integrate sampling at corner points of undecided regions to estimate approximation error and improve classification accuracy.

Experimental results

Research questions

• RQ1 Can a parameter synthesis technique be developed that scales to models with millions of states and dozens of dependent parameters, while remaining sound and efficient?
• RQ2 Is it possible to avoid symbolic computation of rational functions and SMT solving by replacing parametric choices with nondeterministic extremal choices?
• RQ3 Can the resulting parameter-free models be verified efficiently using off-the-shelf numerical algorithms to yield tight, refinable bounds on the parameter space?
• RQ4 How does the coverage of the parameter space (safe/unsafe regions) compare to existing SMT-based and sampling-based approaches in practice?
• RQ5 To what extent does the method’s performance depend on the number of parameters, state space size, and graph structure?

Key findings

• The parameter lifting approach achieves a parameter space coverage of at least 95% within four minutes for two-parameter models, significantly outperforming existing techniques.
• For two-parameter benchmarks, the method classified over 94% of the parameter space as safe or unsafe in under four minutes, with minimal runtime dependence on the number of parameters.
• The approach scales to models with up to millions of states and thousands of parametric transitions, demonstrating orders-of-magnitude speedup over state elimination and SMT-based methods.
• The method provides the first sound and feasible approach for parameter synthesis in parametric MDPs, overcoming the limitations of prior unsound heuristics.
• Runtime is primarily influenced by the number of states and graph structure, not the number of parameters, making it suitable for high-dimensional parameter spaces when combined with smart region selection.
• The approximation error is low and primarily due to region selection rather than the method’s inherent over-approximation, as confirmed by sampling at corner points.