[Paper Review] Game-Theoretic Multiagent Reinforcement Learning

Yaodong Yang, Ma, Chengdong+5 more|arXiv (Cornell University)|Nov 1, 2020

Reinforcement Learning in Robotics398 references146 citations

TL;DR

This paper provides a self-contained overview of multi-agent reinforcement learning (MARL) from a game-theoretical viewpoint, detailing fundamentals (stochastic and extensive-form games) and surveying recent algorithmic advances across various MARL settings.

ABSTRACT

Tremendous advances have been made in multiagent reinforcement learning (MARL). MARL corresponds to the learning problem in a multiagent system in which multiple agents learn simultaneously. It is an interdisciplinary field of study with a long history that includes game theory, machine learning, stochastic control, psychology, and optimization. Despite great successes in MARL, there is a lack of a self-contained overview of the literature that covers game-theoretic foundations of modern MARL methods and summarizes the recent advances. The majority of existing surveys are outdated and do not fully cover the recent developments since 2010. In this work, we provide a monograph on MARL that covers both the fundamentals and the latest developments on the research frontier. The goal of this monograph is to provide a self-contained assessment of the current state-of-the-art MARL techniques from a game-theoretic perspective. We expect this work to serve as a stepping stone for both new researchers who are about to enter this fast-growing field and experts in the field who want to obtain a panoramic view and identify new directions based on recent advances.

Motivation & Objective

• Introduce MARL problem formulations through stochastic games and extensive-form games.
• Explain solution concepts such as Nash equilibrium and policy/value-based approaches in MARL.
• Survey recent MARL algorithmic developments and organize them into coherent taxonomies.
• Discuss grand challenges in MARL, including complexity, non-stationarity, and scalability.
• Highlight modern topics like mean-field MARL and general-sum versus zero-sum settings.

Proposed method

• Present two representative MARL frameworks: stochastic games and extensive-form games.
• Describe value-based and policy-based MARL methods in the multi-agent context.
• Discuss the Nash equilibrium as a solution concept for MARL.
• Introduce special SG types (single-controller, SR-SIT) with tractability notes.
• Survey recent MARL surveys to build a taxonomy of methods.
• Cover modern topics such as Q-function factorisation, multi-agent soft learning, mean-field MARL, and online MDPs.

Experimental results

Research questions

• RQ1 What are the fundamental game-theoretic formulations used to model MARL?
• RQ2 What are the main algorithmic families for solving MARL under stochastic and extensive-form game frameworks?
• RQ3 How do recent advances address challenges like non-stationarity, scalability, and multi-objective learning in MARL?
• RQ4 How can MARL be categorized into zero-sum, general-sum, and mean-field settings, and what methods suit each category?
• RQ5 What are open directions and future research opportunities derived from current MARL surveys?

Key findings

• The paper provides a structured, self-contained treatment of MARL from game theory, bridging fundamentals and modern methods.
• It covers both stochastic games and extensive-form games as core MARL formulations and discusses solution concepts like Nash equilibria.
• It identifies and explains critical MARL challenges such as combinatorial complexity, non-stationarity, and scalability with many agents.
• It surveys a wide range of algorithmic approaches, including value-based, policy-based, and actor-critic methods, in multi-agent settings.
• It introduces advanced topics (mean-field MARL, stochastic potential games, online MDPs) and discusses their implications for future research.