[Paper Review] Multi-objective Robust Strategy Synthesis for Interval Markov Decision Processes

Ernst Moritz Hahn, Vahid Hashemi+3 more|arXiv (Cornell University)|Jun 21, 2017

Formal Methods in Verification21 references4 citations

TL;DR

This paper proposes a value iteration-based algorithm for multi-objective robust strategy synthesis in interval Markov decision processes (IMDPs), where transition probabilities are uncertain and represented as intervals. It addresses the PSPACE-hard problem of finding strategies that robustly satisfy multiple objectives under uncertainty, demonstrating practical effectiveness through case studies using a prototype tool.

ABSTRACT

Interval Markov decision processes (IMDPs) generalise classical MDPs by having interval-valued transition probabilities. They provide a powerful modelling tool for probabilistic systems with an additional variation or uncertainty that prevents the knowledge of the exact transition probabilities. In this paper, we consider the problem of multi-objective robust strategy synthesis for interval MDPs, where the aim is to find a robust strategy that guarantees the satisfaction of multiple properties at the same time in face of the transition probability uncertainty. We first show that this problem is PSPACE-hard. Then, we provide a value iteration-based decision algorithm to approximate the Pareto set of achievable points. We finally demonstrate the practical effectiveness of our proposed approaches by applying them on several case studies using a prototypical tool.

Motivation & Objective

• Address the challenge of synthesizing strategies that robustly satisfy multiple objectives in the presence of uncertain transition probabilities in MDPs.
• Formalize the problem of multi-objective robust strategy synthesis in interval MDPs as a PSPACE-hard decision problem.
• Develop an algorithmic approach to approximate the Pareto set of achievable performance points under model uncertainty.
• Ensure the synthesized strategies are robust across all possible instantiations of interval-valued transition probabilities.
• Demonstrate the practical applicability of the method through case studies using a prototype implementation.

Proposed method

• Formulate interval MDPs as a generalization of classical MDPs, where transition probabilities are specified as intervals rather than exact values.
• Define robust satisfaction of multiple objectives as requiring that all objectives are met for every possible instantiation of the interval probabilities.
• Design a value iteration-based algorithm that incrementally computes lower and upper bounds on the achievable probabilities for each objective.
• Maintain and refine a set of candidate strategies that are robust across the uncertainty domain.
• Use interval arithmetic and convex optimization techniques to propagate uncertainty through the value iteration process.
• Construct a Pareto set approximation by identifying non-dominated points in the multi-objective outcome space under robustness constraints.

Experimental results

Research questions

• RQ1 What is the computational complexity of multi-objective robust strategy synthesis in interval MDPs?
• RQ2 How can one efficiently approximate the Pareto set of achievable objectives under model uncertainty in interval MDPs?
• RQ3 Can a value iteration-based algorithm be adapted to handle robustness against interval-valued transition probabilities in multi-objective settings?
• RQ4 What are the practical performance characteristics of the proposed method on real-world case studies?
• RQ5 How do the robust strategies synthesized by the method compare to non-robust counterparts in terms of reliability under uncertainty?

Key findings

• The problem of multi-objective robust strategy synthesis in interval MDPs is proven to be PSPACE-hard, indicating high computational complexity.
• The proposed value iteration-based algorithm effectively approximates the Pareto set of achievable objectives under robustness constraints.
• The method successfully computes strategies that satisfy multiple objectives across all possible instantiations of interval transition probabilities.
• Case studies demonstrate the practical feasibility and effectiveness of the approach using a prototype tool.
• The algorithm maintains computational tractability by iteratively refining bounds and pruning dominated strategies.
• The results show that robust strategies can be synthesized even when transition probabilities are only known within intervals, ensuring reliability under uncertainty.