[论文解读] Strategy Improvement for Concurrent Reachability and Safety Games

Krishnendu Chatterjee, Luca de Alfaro+1 more|arXiv (Cornell University)|Jan 13, 2012

Formal Methods in Verification参考文献 24被引用 4

一句话总结

本文提出了一类用于并发可达性与安全博弈的策略改进算法，实现了从两个方向单调收敛至博弈值。该工作首次提出了通过迭代改进实现无记忆ε-最优策略的实用方法，解决了先前值迭代方法仅能单向收敛且缺乏见证策略的局限性。

ABSTRACT

We consider concurrent games played on graphs. At every round of a game, each player simultaneously and independently selects a move; the moves jointly determine the transition to a successor state. Two basic objectives are the safety objective to stay forever in a given set of states, and its dual, the reachability objective to reach a given set of states. First, we present a simple proof of the fact that in concurrent reachability games, for all $ε>0$, memoryless $ε$-optimal strategies exist. A memoryless strategy is independent of the history of plays, and an $ε$-optimal strategy achieves the objective with probability within $ε$ of the value of the game. In contrast to previous proofs of this fact, our proof is more elementary and more combinatorial. Second, we present a strategy-improvement (a.k.a.\ policy-iteration) algorithm for concurrent games with reachability objectives. We then present a strategy-improvement algorithm for concurrent games with safety objectives. Our algorithms yield sequences of player-1 strategies which ensure probabilities of winning that converge monotonically to the value of the game. Our result is significant because the strategy-improvement algorithm for safety games provides, for the first time, a way to approximate the value of a concurrent safety game from below. Previous methods could approximate the values of these games only from one direction, and as no rates of convergence are known, they did not provide a practical way to solve these games.

研究动机与目标

• 开发一种策略改进算法，使并发可达性博弈的值从下方单调收敛。
• 设计一种用于并发安全博弈的策略改进算法，从下方逼近博弈值，填补现有方法的空白。
• 通过组合论证证明无记忆ε-最优策略的存在性，避免依赖复杂的不动点定理。
• 提供一种实用且高效的近似方法，用于并发博弈中的ε-最优策略，克服穷举搜索与值迭代的局限性。
• 基于双重估值序列建立终止条件，保证在有界迭代次数内实现ε-近似。

提出的方法

• 为并发可达性博弈提出一种策略改进算法，生成单调非递减的玩家1估值序列，收敛至博弈值。
• 引入一种双重策略改进算法用于安全博弈，通过改进玩家2的策略，使玩家1的安全目标估值序列单调非递减。
• 采用一对双重估值序列 (u_i, v_i) 分别估计安全目标与可达性目标的值，其中 u_i 估计安全值，v_i 估计可达性值。
• 使用条件 u_i + v_i ≥ 1 - ε 作为停止条件，确保同时对两个博弈值实现ε-近似。
• 应用 k-均匀无记忆策略以限制搜索空间，实现有限收敛，且迭代次数具有双指数界。
• 通过 k-均匀策略抽象将并发博弈约化为轮流进行的随机博弈，利用后者已知的终止界。

实验结果

研究问题

• RQ1 能否为并发安全博弈设计一种策略改进算法，使其从下方单调收敛至博弈值？
• RQ2 是否存在一种组合论证，证明并发可达性博弈中无记忆ε-最优策略的存在性，且不依赖不动点或线性规划方法？
• RQ3 策略改进算法能否比穷举搜索更高效地近似并发博弈中的ε-最优策略？
• RQ4 何种终止条件可保证使用双重估值序列实现博弈值的ε-近似？
• RQ5 策略改进算法在并发博弈中的计算复杂度是多少？与值迭代和穷举搜索相比如何？

主要发现

• 为并发可达性博弈中无记忆ε-最优策略的存在性提供了新的、基础的组合论证。
• 所提出的并发可达性博弈策略改进算法生成单调非递减的估值序列，收敛至博弈值。
• 对于并发安全博弈，该算法是首个从下方逼近博弈值的方法，可实现ε-近似策略的实用计算。
• 当 u_i + v_i ≥ 1 - ε 时算法终止，确保 u_i 和 v_i 均在ε范围内接近真实博弈值，且满足 u_i ≤ ⟨1⟩_val(Safe(F)) ≤ 1 - u_i。
• 迭代次数在博弈规模上具有双指数界，根据近期结果，此界接近最优。
• 该方法提供 k-均匀无记忆的见证策略，其中 k 被限制在 (1/ε)^{2^{O(|G|)}} 范围内，实现高效的ε-近似。