[Paper Review] Strategy iteration is strongly polynomial for 2-player turn-based stochastic games with a constant discount factor

Thomas Dueholm Hansen, Peter Bro Miltersen+1 more|arXiv (Cornell University)|Aug 3, 2010

Auction Theory and Applications13 citations

TL;DR

This paper proves that the strategy iteration algorithm for 2-player turn-based stochastic games (2TBSGs) with a constant discount factor runs in strongly polynomial time, specifically O(m/(1−γ) log(n/(1−γ))) iterations, where m is the number of actions and n the number of states. The key contribution is the first strongly polynomial algorithm for solving 2TBSGs, resolving a long-standing open problem by extending Ye's analysis of policy iteration in MDPs to the two-player setting using novel bounds on strategy improvement and value vector convergence.

ABSTRACT

Ye showed recently that the simplex method with Dantzig pivoting rule, as well as Howard's policy iteration algorithm, solve discounted Markov decision processes (MDPs), with a constant discount factor, in strongly polynomial time. More precisely, Ye showed that both algorithms terminate after at most $O(\frac{mn}{1-γ}\log(\frac{n}{1-γ}))$ iterations, where $n$ is the number of states, $m$ is the total number of actions in the MDP, and $0

Motivation & Objective

• To resolve the long-standing open problem of whether strategy iteration for 2TBSGs with a constant discount factor is strongly polynomial.
• To extend Ye's analysis of Howard's policy iteration in MDPs to the two-player setting of 2TBSGs.
• To establish a tight upper bound on the number of iterations required by the strategy iteration algorithm for 2TBSGs.
• To demonstrate that strategy iteration is not only polynomial but strongly polynomial when the discount factor γ is constant.
• To provide a theoretical foundation for future polynomial-time algorithms for non-discounted 2TBSGs and related games such as Simple Stochastic Games and Parity Games.

Proposed method

• Leverages the relationship between strategy iteration and value iteration to bound convergence speed.
• Uses a novel lemma (Lemma 5.5) to show geometric decay of the distance between current and optimal value vectors: ||vk − v*||∞ ≤ γk ||v0 − v*||∞.
• Applies a key technical tool (Lemma 6.4) showing that every L = log(1/γ)(n²/(1−γ)) iterations, a new action appears in the strategy sequence that was not used before.
• Employs a potential function argument based on the 1-norm difference between value vectors to bound improvement per iteration.
• Uses the fact that the optimal strategy is positional and that strategy iteration monotonically improves value vectors to ensure convergence.
• Combines bounds on value vector decay and action diversity to derive the final iteration count via a pigeonhole-style argument on the number of distinct actions.

Experimental results

Research questions

• RQ1 Can the strategy iteration algorithm for 2TBSGs be proven to run in strongly polynomial time when the discount factor γ is constant?
• RQ2 Does the iteration bound for Howard’s policy iteration in MDPs extend to the two-player generalization, i.e., strategy iteration for 2TBSGs?
• RQ3 What is the tightest possible upper bound on the number of iterations for strategy iteration in 2TBSGs with constant γ?
• RQ4 Can the analysis of value vector convergence be leveraged to prove a strongly polynomial bound in the two-player setting?
• RQ5 Is there a structural property of the strategy sequence (e.g., appearance of new actions) that can be used to bound the number of iterations?

Key findings

• The strategy iteration algorithm for 2TBSGs with a constant discount factor γ terminates in at most O(m/(1−γ) log(n/(1−γ))) iterations.
• This bound improves upon Ye’s earlier bound for MDPs by a factor of n, reducing it from O(mn/(1−γ) log(n/(1−γ))) to O(m/(1−γ) log(n/(1−γ))).
• The bound is strongly polynomial when γ is constant, as the dependence on n and m is polynomial and the logarithmic term depends only on n and γ.
• The algorithm guarantees convergence to an optimal positional strategy for both players, leveraging the existence of positional optimal strategies in 2TBSGs.
• The proof establishes that every L = log(1/γ)(n²/(1−γ)) iterations, a new action appears in the strategy sequence, and since there are only m actions, the total number of such steps is bounded by (m+1)L.
• The result resolves a long-standing open problem by providing the first strongly polynomial algorithm for solving 2TBSGs with a constant discount factor.