[Paper Review] Approximate Fictitious Play for Mean Field Games

Romuald Élie, Julien Pérolat+3 more|arXiv (Cornell University)|Jul 4, 2019

Experimental Behavioral Economics Studies21 references19 citations

TL;DR

This paper introduces an approximate fictitious play framework for Mean Field Games (MFG) where agents learn their best response via model-free reinforcement learning (RL), enabling convergence to approximate non-stationary Nash equilibria under standard MFG dynamics. It establishes the first theoretical convergence of model-free RL algorithms to such equilibria in continuous action spaces.

ABSTRACT

The theory of Mean Field Games (MFG) allows characterizing the Nash equilibria of an infinite number of identical players, and provides a convenient and relevant mathematical framework for the study of games with a large number of agents in interaction. Until very recently, the literature only considered Nash equilibria between fully informed players. In this paper, we focus on the realistic setting where agents with no prior information on the game learn their best response policy through repeated experience. We study the convergence to a (possibly approximate) Nash equilibrium of a fictitious play iterative learning scheme where the best response is approximately computed, typically by a reinforcement learning (RL) algorithm. Notably, we show for the first time convergence of model free learning algorithms towards non-stationary MFG equilibria, relying only on classical assumptions on the MFG dynamics. We illustrate our theoretical results with a numerical experiment in continuous action-space setting, where the best response of the iterative fictitious play scheme is computed with a deep RL algorithm.

Motivation & Objective

• To model realistic learning scenarios in Mean Field Games where agents lack prior knowledge and learn through repeated interactions.
• To analyze the convergence of iterative fictitious play when best responses are approximated via reinforcement learning.
• To extend theoretical convergence results to non-stationary MFG equilibria using only classical assumptions on MFG dynamics.
• To demonstrate the feasibility of model-free RL in continuous action-space MFG settings through numerical validation.

Proposed method

• Employs an iterative fictitious play scheme where agents update their policies based on empirical frequency of others' actions.
• Approximates the best response policy using a model-free reinforcement learning algorithm, such as deep RL.
• Introduces a learning rule that updates agent policies based on historical action frequencies and approximate best responses.
• Relies on standard assumptions about the MFG dynamics, including Lipschitz continuity and regularity of the value function.
• Uses function approximation (e.g., deep neural networks) to represent policies and value functions in continuous action spaces.
• Employs a two-timescale learning update to stabilize policy and value function estimation.

Experimental results

Research questions

• RQ1 Can fictitious play with approximate best responses converge to a Nash equilibrium in Mean Field Games?
• RQ2 Does model-free reinforcement learning converge to non-stationary MFG equilibria under classical MFG assumptions?
• RQ3 How does the convergence behavior change when best responses are computed via deep RL in continuous action spaces?
• RQ4 What are the theoretical guarantees for learning in large-population games with uninformed agents?

Key findings

• The paper proves convergence of the approximate fictitious play scheme to a (possibly approximate) Nash equilibrium in Mean Field Games.
• Convergence is established under standard assumptions on MFG dynamics, including Lipschitz continuity and smoothness.
• The framework successfully supports learning in continuous action spaces using deep reinforcement learning for best response approximation.
• Numerical experiments confirm convergence and stability of the learning dynamics in a continuous action-space MFG setting.
• The results represent the first theoretical convergence guarantee for model-free RL algorithms in non-stationary MFG equilibria under classical assumptions.
• The approach enables learning without prior knowledge of the game's structure, making it suitable for real-world large-scale multi-agent systems.