[Paper Review] Independent Learning in Stochastic Games

Asuman Ozdaglar, Muhammed O. Sayin+1 more|arXiv (Cornell University)|Nov 23, 2021

Advanced Bandit Algorithms Research71 references4 citations

TL;DR

This paper proposes novel independent learning dynamics for zero-sum stochastic games that guarantee convergence without coordination between agents. By extending fictitious play to dynamic environments using best-response updates with asymmetric stepsizes, the authors achieve convergence in model-based, model-free, and minimal-information settings, offering a decentralized solution to multi-agent reinforcement learning in non-stationary environments.

ABSTRACT

Reinforcement learning (RL) has recently achieved tremendous successes in many artificial intelligence applications. Many of the forefront applications of RL involve multiple agents, e.g., playing chess and Go games, autonomous driving, and robotics. Unfortunately, the framework upon which classical RL builds is inappropriate for multi-agent learning, as it assumes an agent's environment is stationary and does not take into account the adaptivity of other agents. In this review paper, we present the model of stochastic games for multi-agent learning in dynamic environments. We focus on the development of simple and independent learning dynamics for stochastic games: each agent is myopic and chooses best-response type actions to other agents' strategy without any coordination with her opponent. There has been limited progress on developing convergent best-response type independent learning dynamics for stochastic games. We present our recently proposed simple and independent learning dynamics that guarantee convergence in zero-sum stochastic games, together with a review of other contemporaneous algorithms for dynamic multi-agent learning in this setting. Along the way, we also reexamine some classical results from both the game theory and RL literature, to situate both the conceptual contributions of our independent learning dynamics, and the mathematical novelties of our analysis. We hope this review paper serves as an impetus for the resurgence of studying independent and natural learning dynamics in game theory, for the more challenging settings with a dynamic environment.

Motivation & Objective

• To address the lack of convergent, independent learning dynamics in stochastic games, where classical RL assumes stationary environments.
• To develop decentralized learning rules that do not require coordination or knowledge of opponent objectives.
• To extend fictitious play-type dynamics to dynamic, non-stationary environments such as stochastic games.
• To establish convergence guarantees under various information settings: model-based, model-free, and minimal information.
• To bridge game theory and reinforcement learning by combining best-response dynamics with stochastic game theory.

Proposed method

• Proposes a fictitious-play-inspired learning rule where each agent updates its strategy based on empirical estimates of opponents' past actions.
• Uses asymmetric stepsizes in updates, with one agent updating faster than the other, to ensure convergence in zero-sum stochastic games.
• Applies the dynamics in three information regimes: full model knowledge, partial model knowledge, and minimal observation of opponent actions.
• Employs a continuation payoff mechanism in continuous-time embedding to maintain zero-sum structure during learning.
• Integrates ideas from game theory and reinforcement learning to handle dynamic transitions and payoff estimation.
• Analyzes convergence using stochastic approximation and Lyapunov-based techniques, ensuring asymptotic convergence to Nash equilibrium.

Experimental results

Research questions

• RQ1 Can independent, best-response-type learning dynamics converge in zero-sum stochastic games without coordination between agents?
• RQ2 How can fictitious play be extended to dynamic environments with non-stationary transitions?
• RQ3 What learning guarantees are possible when agents have limited or no information about the opponent’s strategy or payoff functions?
• RQ4 Can decentralized learning dynamics achieve convergence in infinite-horizon discounted stochastic games?
• RQ5 What role do asymmetric update speeds play in stabilizing learning in competitive, dynamic settings?

Key findings

• The proposed independent learning dynamics converge to a Nash equilibrium in zero-sum stochastic games under all three information settings: model-based, model-free, and minimal information.
• Convergence is achieved without requiring agents to know the opponent’s objective or even to observe the opponent’s actions in the minimal-information setting.
• The use of asymmetric stepsizes enables convergence, unlike symmetric or coordinated update rules that may fail to converge.
• The method provides the first convergence guarantee for fully decentralized, independent learning in stochastic games, filling a long-standing gap in the literature.
• The analysis extends to continuous-time embeddings and establishes convergence via a time-averaged continuation payoff mechanism that preserves the zero-sum structure.
• The framework is robust to model uncertainty and supports learning in environments with unknown transition probabilities and payoff functions.