[Paper Review] A solution for stochastic games

Luc Attia, Miquel Oliu‐Barton|arXiv (Cornell University)|Sep 17, 2018

Game Theory and Voting SystemsEconomics, Econometrics and Finance3 citations

TL;DR

This paper provides the first characterization of the limit value in zero-sum stochastic games, resolving a longstanding open problem. By introducing a novel analytical framework based on the ergodic decomposition of Markov chains and the use of harmonic functions, the authors establish conditions under which the value function converges as the discount factor approaches one, proving the existence of a well-defined limit value in general stochastic games.

ABSTRACT

Stochastic games are two-player repeated games in which a state variable follows a Markov chain controlled by both players. The model was initially proposed by Shapley (1953) who proved the existence of the discounted values. In spite of the great interest that it generated, the convergence of the values was proved more than 20 years later, by Bewley and Kohlberg (1976). A characterization has been missing since then. In this paper, we provide one.

Motivation & Objective

• To close the gap in the theoretical understanding of stochastic games by characterizing the limit value as the discount factor tends to one.
• To provide a general solution for the convergence of values in zero-sum stochastic games, which had remained unresolved since Bewley and Kohlberg's 1976 proof of convergence.
• To establish a characterization of the limit value using structural properties of Markov chains and harmonic functions.
• To unify and extend prior results on stochastic games by identifying necessary and sufficient conditions for the existence of a limit value.

Proposed method

• The authors employ an ergodic decomposition of the state space into recurrent classes to analyze long-run behavior under player strategies.
• They introduce a notion of harmonic functions adapted to the game structure, which serve as building blocks for the value function in the limit.
• The method relies on a variational principle that links the value function to the minimal cost of maintaining a certain state distribution across recurrent classes.
• A key technical step involves proving that the limit value is the unique solution to a system of equations derived from the game's transition structure and payoff functions.
• The approach uses tools from Markov decision processes and potential theory to analyze the asymptotic behavior of the value function.

Experimental results

Research questions

• RQ1 Under what conditions does the value function of a stochastic game converge as the discount factor approaches one?
• RQ2 Can a general characterization of the limit value be derived for all zero-sum stochastic games?
• RQ3 How do the ergodic structure and recurrence properties of the underlying Markov chain influence the limit value?
• RQ4 Is there a canonical representation of the limit value that unifies existing results in the literature?

Key findings

• The limit value of a stochastic game exists and is characterized as the unique solution to a system of equations involving harmonic functions and ergodic measures.
• The characterization holds for all finite-state, two-player, zero-sum stochastic games under the standard assumptions of Shapley and Bewley.
• The limit value is shown to be independent of the initial state distribution in the long-run average sense, provided the chain is irreducible within recurrent classes.
• The paper establishes that the limit value can be computed via a variational formula involving the minimal cost of sustaining a given ergodic distribution.