[Paper Review] Stochastic Potential Games

David Mguni|arXiv (Cornell University)|May 27, 2020

Game Theory and Applications38 references4 citations

TL;DR

This paper introduces stochastic potential games (SPGs), a tractable subclass of N-player non-zero-sum stochastic games where Markov perfect Nash equilibria (MP-NE) can be computed in polynomial time. By leveraging a dual Markov decision process, the method enables efficient computation of equilibria even for more than two players, overcoming the PSPACE-hard complexity of general stochastic games.

ABSTRACT

Computing the Nash equilibrium (NE) for N-player non-zerosum stochastic games is a formidable challenge. Currently, algorithmic methods in stochastic game theory are unable to compute NE for stochastic games (SGs) for settings in all but extreme cases in which the players either play as a team or have diametrically opposed objectives in a two-player setting. This greatly impedes the application of the SG framework to numerous problems within economics and practical systems of interest. In this paper, we provide a method of computing Nash equilibria in nonzero-sum settings and for populations of players more than two. In particular, we identify a subset of SGs known as stochastic potential games (SPGs) for which the (Markov perfect) Nash equilibrium can be computed tractably and in polynomial time. Unlike SGs for which, in general, computing the NE is PSPACE-hard, we show that SGs with the potential property are P-Complete. We further demonstrate that for each SPG there is a dual Markov decision process whose solution coincides with the MP-NE of the SPG. We lastly provide algorithms that tractably compute the MP-NE for SGs with more than two players.

Motivation & Objective

• To address the computational intractability of finding Nash equilibria in general N-player stochastic games, which are typically PSPACE-hard.
• To identify a tractable subclass of stochastic games—stochastic potential games (SPGs)—where Nash equilibria can be computed efficiently.
• To demonstrate that SPGs admit a dual Markov decision process whose solution yields the Markov perfect Nash equilibrium (MP-NE).
• To provide scalable algorithms for computing MP-NE in stochastic games with more than two players.

Proposed method

• Identify the potential property in stochastic games, ensuring that the game's payoff structure aligns with a potential function that aggregates player incentives.
• Prove that SPGs are P-Complete, meaning they can be solved in polynomial time, in contrast to the general PSPACE-hard complexity of stochastic games.
• Construct a dual Markov decision process (MDP) whose optimal policy corresponds exactly to the MP-NE of the SPG.
• Leverage the duality between the SPG and its dual MDP to transform the equilibrium computation into a single-agent planning problem.
• Design scalable algorithms that compute the MP-NE by solving the dual MDP, enabling application to games with more than two players.
• Ensure the solution method is computationally efficient and runs in polynomial time, making it feasible for practical systems.

Experimental results

Research questions

• RQ1 Can Nash equilibria in N-player non-zero-sum stochastic games be computed efficiently beyond the special cases of team play or two-player zero-sum games?
• RQ2 What structural properties of stochastic games allow for polynomial-time computation of Nash equilibria?
• RQ3 Is there a dual Markov decision process that can be used to compute the Markov perfect Nash equilibrium of a stochastic potential game?
• RQ4 How does the potential function in SPGs relate to the existence and computation of equilibria in multi-player settings?

Key findings

• Stochastic potential games (SPGs) are a subclass of stochastic games for which computing the Markov perfect Nash equilibrium (MP-NE) is tractable and solvable in polynomial time.
• Unlike general stochastic games, which are PSPACE-hard, SPGs are P-Complete, indicating a significant reduction in computational complexity.
• For every SPG, there exists a dual Markov decision process whose solution yields the MP-NE of the original game.
• The proposed method enables the computation of MP-NE for stochastic games with more than two players, extending beyond the scope of prior tractable methods.
• The duality between the SPG and its dual MDP allows the equilibrium computation to be reduced to a single-agent optimization problem.
• The algorithms developed in the paper provide a scalable and efficient approach to computing equilibria in complex, multi-agent stochastic environments.