[Paper Review] Mean Field Multi-Agent Reinforcement Learning

Yaodong Yang, Rui Luo+4 more|arXiv (Cornell University)|Feb 15, 2018

Game Theory and Applications50 references246 citations

TL;DR

This paper introduces mean field approximations to scale multi-agent reinforcement learning to many agents, proposing MF-Q and MF-AC algorithms with convergence to Nash equilibrium under certain assumptions, and validates them on Gaussian squeeze, Ising, and battle games.

ABSTRACT

Existing multi-agent reinforcement learning methods are limited typically to a small number of agents. When the agent number increases largely, the learning becomes intractable due to the curse of the dimensionality and the exponential growth of agent interactions. In this paper, we present \emph{Mean Field Reinforcement Learning} where the interactions within the population of agents are approximated by those between a single agent and the average effect from the overall population or neighboring agents; the interplay between the two entities is mutually reinforced: the learning of the individual agent's optimal policy depends on the dynamics of the population, while the dynamics of the population change according to the collective patterns of the individual policies. We develop practical mean field Q-learning and mean field Actor-Critic algorithms and analyze the convergence of the solution to Nash equilibrium. Experiments on Gaussian squeeze, Ising model, and battle games justify the learning effectiveness of our mean field approaches. In addition, we report the first result to solve the Ising model via model-free reinforcement learning methods.

Motivation & Objective

• Address the scalability challenge in MARL when the number of agents is large.
• Introduce a mean field framework to approximate interactions between an agent and the population.
• Develop and analyze practical mean field Q-learning and mean field actor-critic algorithms.
• Establish convergence guarantees to Nash equilibrium under stated assumptions.
• Demonstrate effectiveness through experiments on Gaussian squeeze, Ising model, and mixed cooperative-competitive games.

Proposed method

• Factorize the multi-agent Q-function to depend on an agent and the mean action from neighbors (Eq. 5).
• Apply mean field approximation to replace pairwise neighbor interactions with a mean field term (Eq. 6–8).
• Update mean field Q-values via a TD-like rule using the mean action ar{a} and a value function v^{j}_{MF} (Eq. 9–10).
• Compute best responses with iterative mean action updates and a Boltzmann policy or an explicit actor-critic network (Eq. 11–12).
• Prove contraction of the mean field operator and convergence of MF-Q to Nash-Q under assumptions (Theorem 1).
• Present MF-Q (off-policy) and MF-AC (on-policy) implementations with neural network function approximators and gradient-based updates (Eq. 14).

Experimental results

Research questions

• RQ1 Can mean field approximations enable scalable learning in MARL with a large population of agents?
• RQ2 Do MF-Q and MF-AC converge to Nash equilibria under standard MARL assumptions?
• RQ3 How do mean field methods perform on representative tasks such as Gaussian squeeze, Ising model, and mixed cooperative-competitive games?

Key findings

• MF-Q and MF-AC scale to many agents by converting many-body interactions into two-agent interactions (agent vs. mean field).
• The mean field operator is a contraction under certain assumptions, ensuring convergence to Nash-Q values.
• Experiments show learning effectiveness in Gaussian squeeze, solving the Ising model without explicit energy function, and achieving high win rates in a mixed battle game.