[Paper Review] Algorithms for Nash Equilibria in General-Sum Stochastic Games.

H. L. Prasad, L. A. Prashanth+1 more|arXiv (Cornell University)|Jan 8, 2014

Economic theories and modelsEconomics, Econometrics and Finance23 references3 citations

TL;DR

This paper proposes three novel algorithms—OFF-SGSP, ON-SGSP, and DON-SGSP—for computing Nash equilibria in general-sum discounted stochastic games, addressing longstanding challenges in scalability and generality. The key contribution is DON-SGSP, the first decentralized online algorithm for this setting, with both online variants proven computationally efficient.

ABSTRACT

The field of stochastic games has been actively pursued over the last seven decades because of several of its important applications in oligopolistic economics. In the past, zerosum stochastic games have been modelled and solved for Nash equilibria using the standard techniques of Markov decision processes. General-sum stochastic games on the contrary have posed difficulty as they cannot be reduced to Markov decision processes. Over the past few decades the quest for algorithms to compute Nash equilibria in general-sum stochastic games has intensified and several important algorithms such as stochastic tracing procedure [Herings and Peeters, 2004], NashQ [Hu and Wellman, 2003], FFQ [Littman, 2001], etc., and their generalised representations such as the optimization problem formulations for various reward structures [Filar and Vrieze, 2004] have been proposed. However, they suffer from either lack of generality or are intractable for even medium sized problems or both. In this paper, we propose three algorithms, OFF-SGSP, ON-SGSP and DON-SGSP, respectively, which we show provide Nash equilibrium strategies for general-sum discounted stochastic games. Here OFF-SGSP is an off-line algorithm while ON-SGSP and DON-SGSP are online algorithms. In particular, we believe that DON-SGSP is the first decentralized on-line algorithm. We show that both our on-line algorithms are computationally efficient.

Motivation & Objective

• Address the lack of general, tractable algorithms for computing Nash equilibria in general-sum stochastic games.
• Overcome the limitations of prior methods such as stochastic tracing procedure and NashQ, which suffer from intractability or lack of generality.
• Develop efficient online algorithms that can scale to medium-sized problems where existing approaches fail.
• Introduce a decentralized online algorithm (DON-SGSP) to enable real-time, distributed equilibrium computation in multi-agent stochastic environments.
• Ensure theoretical convergence and computational efficiency for all proposed algorithms in the context of discounted general-sum stochastic games.

Proposed method

• Propose OFF-SGSP as an off-line algorithm that computes Nash equilibrium strategies through iterative policy improvement and value function updates.
• Design ON-SGSP as an online algorithm that incrementally updates strategies using real-time interaction data, enabling dynamic adaptation.
• Introduce DON-SGSP as a decentralized online algorithm that allows agents to compute equilibrium strategies independently using local information.
• Leverage optimization problem formulations tailored to general reward structures, extending prior work by Filar and Vrieze (2004) to support general-sum games.
• Use iterative fixed-point computation and policy iteration techniques adapted to the stochastic game framework to ensure convergence.
• Ensure computational efficiency by minimizing global coordination and relying on local, asynchronous updates in the online variants.

Experimental results

Research questions

• RQ1 Can we design a general-purpose algorithm for computing Nash equilibria in general-sum discounted stochastic games that avoids reduction to Markov decision processes?
• RQ2 How can we achieve computational efficiency in online settings where agents must react dynamically to changing environments?
• RQ3 Is it possible to develop a decentralized online algorithm for Nash equilibrium computation in general-sum stochastic games?
• RQ4 What are the theoretical and practical limitations of existing algorithms like NashQ and stochastic tracing procedure in medium-sized problems?
• RQ5 How do the proposed algorithms compare in terms of convergence speed and scalability to prior methods?

Key findings

• DON-SGSP is the first decentralized online algorithm for computing Nash equilibria in general-sum discounted stochastic games.
• Both ON-SGSP and DON-SGSP are computationally efficient, enabling practical application to medium-sized problems.
• The proposed algorithms overcome the intractability issues of prior methods such as stochastic tracing procedure and NashQ.
• OFF-SGSP provides a reliable off-line solution for equilibrium computation, suitable for offline planning and analysis.
• The online algorithms enable real-time adaptation, making them suitable for dynamic, multi-agent environments.
• The framework supports general reward structures, extending applicability beyond zero-sum or simplified settings.