[Paper Review] Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms

Kaiqing Zhang, Zhuoran Yang+1 more|arXiv (Cornell University)|Nov 24, 2019

Innovation Diffusion and Forecasting124 citations

TL;DR

A theoretical survey of MARL focusing on two frameworks (Markov/stochastic games and extensive-form games) with analysis of convergence, complexity, and new angles for future research.

ABSTRACT

Recent years have witnessed significant advances in reinforcement learning (RL), which has registered great success in solving various sequential decision-making problems in machine learning. Most of the successful RL applications, e.g., the games of Go and Poker, robotics, and autonomous driving, involve the participation of more than one single agent, which naturally fall into the realm of multi-agent RL (MARL), a domain with a relatively long history, and has recently re-emerged due to advances in single-agent RL techniques. Though empirically successful, theoretical foundations for MARL are relatively lacking in the literature. In this chapter, we provide a selective overview of MARL, with focus on algorithms backed by theoretical analysis. More specifically, we review the theoretical results of MARL algorithms mainly within two representative frameworks, Markov/stochastic games and extensive-form games, in accordance with the types of tasks they address, i.e., fully cooperative, fully competitive, and a mix of the two. We also introduce several significant but challenging applications of these algorithms. Orthogonal to the existing reviews on MARL, we highlight several new angles and taxonomies of MARL theory, including learning in extensive-form games, decentralized MARL with networked agents, MARL in the mean-field regime, (non-)convergence of policy-based methods for learning in games, etc. Some of the new angles extrapolate from our own research endeavors and interests. Our overall goal with this chapter is, beyond providing an assessment of the current state of the field on the mark, to identify fruitful future research directions on theoretical studies of MARL. We expect this chapter to serve as continuing stimulus for researchers interested in working on this exciting while challenging topic.

Motivation & Objective

• Clarify the theoretical foundations of MARL across representative frameworks (Markov/stochastic games and extensive-form games).
• Organize MARL algorithms with convergence and complexity analyses under fully cooperative, fully competitive, and mixed settings.
• Highlight new angles and taxonomies in MARL theory to guide future research and applications.

Proposed method

• Review and synthesize MARL algorithms with theoretical guarantees within Markov/stochastic and extensive-form game frameworks.
• Discuss challenges such as non-stationarity, joint action space, and information structure, and relate them to equilibrium concepts.
• Introduce and compare settings (cooperative, competitive, mixed) and their implications for learning dynamics and convergence.
• Highlight extensions like decentralized MARL, mean-field MARL, and learning in extensive-form games.

Experimental results

Research questions

• RQ1 What MARL algorithms have theoretical convergence and complexity guarantees under Markov/stochastic and extensive-form game frameworks?
• RQ2 How do cooperative, competitive, and mixed settings affect learning dynamics and equilibrium concepts in MARL?
• RQ3 What are the emerging angles and taxonomies in MARL theory that can guide future theoretical work?

Key findings

• The chapter provides a selective overview of MARL theories and algorithms with emphasis on frameworks and convergence analyses.
• It discusses challenges common to MARL such as non-stationarity, combinatorial joint action spaces, and information structure, and connects them to equilibrium concepts.
• It extends discussion to extensive-form games, decentralized MARL, mean-field MARL, and the (non-)convergence of policy-based methods in zero-sum games.
• It positions Nash equilibrium and ε-Nash equilibrium as central solution concepts in Markov and extensive-form game MARL settings.
• It distinguishes cooperative, competitive, and mixed settings and shows how they map to MGs and extensive-form games, guiding algorithm design.