[Paper Review] Strategy Improvement for Concurrent Safety Games

Krishnendu Chatterjee, Luca de Alfaro+1 more|ArXiv.org|Apr 29, 2008

Formal Methods in Verification18 references3 citations

TL;DR

This paper presents the first strategy improvement algorithm for concurrent safety games, enabling monotonic convergence to the value of the game from below via memoryless randomized strategies. The method provides converging lower and upper bounds when combined with value iteration or reachability strategy improvement, enabling practical computation of game values with error estimation.

ABSTRACT

We consider concurrent games played on graphs. At every round of the game, each player simultaneously and independently selects a move; the moves jointly determine the transition to a successor state. Two basic objectives are the safety objective: ``stay forever in a set F of states'', and its dual, the reachability objective, ``reach a set R of states''. We present in this paper a strategy improvement algorithm for computing the value of a concurrent safety game, that is, the maximal probability with which player 1 can enforce the safety objective. The algorithm yields a sequence of player-1 strategies which ensure probabilities of winning that converge monotonically to the value of the safety game. The significance of the result is twofold. First, while strategy improvement algorithms were known for Markov decision processes and turn-based games, as well as for concurrent reachability games, this is the first strategy improvement algorithm for concurrent safety games. Second, and most importantly, the improvement algorithm provides a way to approximate the value of a concurrent safety game from below (the known value-iteration algorithms approximate the value from above). Thus, when used together with value-iteration algorithms, or with strategy improvement algorithms for reachability games, our algorithm leads to the first practical algorithm for computing converging upper and lower bounds for the value of reachability and safety games.

Motivation & Objective

• To address the lack of practical algorithms for computing converging upper and lower bounds on the value of concurrent safety and reachability games.
• To develop a strategy improvement algorithm that converges monotonically from below toward the true value of concurrent safety games.
• To enable the combination with existing value-iteration or reachability strategy improvement methods for bidirectional convergence and error estimation.
• To establish a foundation for practical, efficient algorithms for solving concurrent safety games with guaranteed convergence and bounds.

Proposed method

• The algorithm generates a sequence of memoryless, randomized strategies for player 1 that monotonically improve the probability of satisfying the safety objective.
• Each strategy is improved by iteratively selecting moves that maximize the worst-case (minimax) probability of staying within the safe set, given player 2's best response.
• The improvement process uses a valuation function that tracks the minimal winning probability over all possible player 2 responses, updated via iterative refinement.
• The method leverages the existence of memoryless optimal strategies for concurrent safety games, ensuring finite convergence to the optimal value.
• The algorithm is designed to be compatible with value-iteration techniques that converge from above, enabling bidirectional bounds on the game value.
• Termination is guaranteed due to the finite number of memoryless strategies and the monotonic improvement in valuation, with bounds derived from rational number representations of probabilities.

Experimental results

Research questions

• RQ1 Can a strategy improvement algorithm be designed for concurrent safety games, given the absence of deterministic optimal strategies?
• RQ2 Does the proposed algorithm ensure monotonic convergence of the winning probability from below toward the true game value?
• RQ3 Can the algorithm be combined with existing value-iteration or reachability strategy improvement methods to yield converging upper and lower bounds?
• RQ4 What are the computational complexity and termination bounds of the strategy improvement process in concurrent safety games?
• RQ5 Is it possible to achieve practical, error-bounded computation of game values using only strategy improvement and value iteration?

Key findings

• The proposed strategy improvement algorithm is the first to provide monotonic convergence from below to the value of concurrent safety games.
• The algorithm generates a sequence of memoryless, randomized strategies for player 1 that converge toward the optimal value, ensuring strictly increasing lower bounds on the winning probability.
• When combined with value-iteration algorithms (which converge from above) or reachability strategy improvement, the method enables the first practical computation of converging upper and lower bounds on game values.
• The algorithm terminates in finite time due to the finite number of memoryless strategies and the monotonic improvement of the valuation function.
• For binary turn-based stochastic games, the algorithm terminates in at most $|S| imes 4^{|S_R|-1}$ steps, and in time $O( ext{min}ig"{\prod_{s o S_1}|E(s)|, 2^{O(|\delta|)}}\big\} \cdot \text{poly}(|G|)$, where $|\delta|$ is the size of the transition function.
• The method establishes a practical framework for solving concurrent safety and reachability games with guaranteed convergence and error estimation, filling a critical gap in algorithmic game theory.