[Paper Review] Optimistic and Topological Value Iteration for Simple Stochastic Games

Muqsit Azeem, Alexandros Evangelidis+3 more|arXiv (Cornell University)|Jul 29, 2022

Economic theories and models4 citations

TL;DR

This paper introduces Optimistic Value Iteration (OVI) and Topological Value Iteration (TVI) for Simple Stochastic Games (SSGs), extending OVI to general SSGs by combining deflation of end components with optimistic guessing of upper bounds. It also proposes a precise, efficient TVI variant that computes exact solutions per strongly connected component (SCC), outperforming state-of-the-art methods on large, deep models with many SCCs.

ABSTRACT

While value iteration (VI) is a standard solution approach to simple stochastic games (SSGs), it suffered from the lack of a stopping criterion. Recently, several solutions have appeared, among them also "optimistic" VI (OVI). However, OVI is applicable only to one-player SSGs with no end components. We lift these two assumptions, making it available to general SSGs. Further, we utilize the idea in the context of topological VI, where we provide an efficient precise solution. In order to compare the new algorithms with the state of the art, we use not only the standard benchmarks, but we also design a random generator of SSGs, which can be biased towards various types of models, aiding in understanding the advantages of different algorithms on SSGs.

Motivation & Objective

• To address the lack of a stopping criterion in value iteration (VI) for general SSGs, which previously only existed for MDPs or restricted SSGs.
• To extend optimistic VI (OVI), which requires no end components, to general SSGs by integrating deflation techniques to handle cyclic dependencies in end components.
• To improve topological VI by enabling precise, fast computation per SCC, overcoming the issue of imprecision propagation from lower to upper SCCs.
• To design a customizable random SSG generator to evaluate algorithm performance across diverse model types and identify strengths and weaknesses.

Proposed method

• Extends OVI to SSGs by combining deflation of 'simple' end components with optimistic guessing of upper bounds, ensuring monotonic convergence.
• Introduces a novel topological VI variant that computes exact solutions within each SCC by guessing optimal strategies rather than values, reducing the number of guesses needed.
• Uses a deflation mechanism to collapse cyclic dependencies in end components, enabling the use of optimistic bounds in SSGs where standard MDP techniques fail.
• Employs a two-phase algorithm: first, VI converges from below; second, optimistic guesses for upper bounds are verified iteratively, with precision adjusted dynamically.
• Designs a random SSG generator biased toward various structural types (e.g., deep chains, many SCCs) to evaluate algorithm scalability and robustness.
• Implements a hybrid approach, PTBVI, combining topological structure with optimistic verification, achieving high performance on large, complex models.

Experimental results

Research questions

• RQ1 Can optimistic value iteration be generalized from one-player MDPs to two-player SSGs with end components, maintaining correctness and convergence?
• RQ2 How can topological value iteration be modified to ensure precise and efficient computation in SSGs, avoiding error propagation across SCCs?
• RQ3 What is the impact of model structure—particularly depth and number of SCCs—on the performance of OVI and TVI variants?
• RQ4 How do the new algorithms compare to state-of-the-art methods like Bounded VI (BVI), Strategy Iteration (SI), and Weighted Policy (WP) on diverse SSG benchmarks?
• RQ5 What parameter choices (e.g., verification phase length, precision halving) yield optimal trade-offs between convergence speed and computational cost?

Key findings

• PTBVI outperforms both TSI-LP and TSI-SI on models with over 2 million states, with a 156-second runtime on simple_1000000_5_SCC compared to 274 seconds for TSI-LP.
• On the simple_5000000_5_SCC model, PTBVI completes in 589.7 seconds, while WP takes 1723.4 seconds, demonstrating superior scalability.
• Topological OVI with strategy guessing reduces the number of required guesses by orders of magnitude compared to value-based optimistic guessing.
• The algorithm scales well on deep models with many SCCs: for simple_5000000_1_SCC, PTBVI runs in 581.999 seconds, outperforming WP (301.517 seconds) on the same model.
• The random SSG generator enables systematic evaluation, revealing that OVI performs well on shallow models but struggles with deep ones, while topological methods excel in deep, modular structures.
• BVI is outperformed by PTBVI on large models due to the high cost of waiting for lower bounds to converge, especially in deep or complex SCC structures.