[Paper Review] On the Approximation of Cooperative Heterogeneous Multi-Agent Reinforcement Learning (MARL) using Mean Field Control (MFC)

Washim Uddin Mondal, Mridul Agarwal+2 more|arXiv (Cornell University)|Sep 8, 2021

Reinforcement Learning in Robotics11 citations

TL;DR

This paper proposes a mean field control (MFC) framework to approximate cooperative heterogeneous multi-agent reinforcement learning (MARL) with K distinct agent classes, proving approximation error bounds that scale with the inverse square root of class sizes. It introduces a Natural Policy Gradient-based algorithm that converges to a near-optimal MARL policy within O(e_j) error with O(e_j^{-3}) sample complexity across three distinct interaction scenarios.

ABSTRACT

Mean field control (MFC) is an effective way to mitigate the curse of dimensionality of cooperative multi-agent reinforcement learning (MARL) problems. This work considers a collection of $N_{\\mathrm{pop}}$ heterogeneous agents that can be segregated into $K$ classes such that the $k$-th class contains $N_k$ homogeneous agents. We aim to prove approximation guarantees of the MARL problem for this heterogeneous system by its corresponding MFC problem. We consider three scenarios where the reward and transition dynamics of all agents are respectively taken to be functions of $(1)$ joint state and action distributions across all classes, $(2)$ individual distributions of each class, and $(3)$ marginal distributions of the entire population. We show that, in these cases, the $K$-class MARL problem can be approximated by MFC with errors given as $e_1=\\mathcal{O}(\\frac{\\sqrt{|\\mathcal{X}|}+\\sqrt{|\\mathcal{U}|}}{N_{\\mathrm{pop}}}\\sum_{k}\\sqrt{N_k})$, $e_2=\\mathcal{O}(\\left[\\sqrt{|\\mathcal{X}|}+\\sqrt{|\\mathcal{U}|}\ ight]\\sum_{k}\\frac{1}{\\sqrt{N_k}})$ and $e_3=\\mathcal{O}\\left(\\left[\\sqrt{|\\mathcal{X}|}+\\sqrt{|\\mathcal{U}|}\ ight]\\left[\\frac{A}{N_{\\mathrm{pop}}}\\sum_{k\\in[K]}\\sqrt{N_k}+\\frac{B}{\\sqrt{N_{\\mathrm{pop}}}}\ ight]\ ight)$, respectively, where $A, B$ are some constants and $|\\mathcal{X}|,|\\mathcal{U}|$ are the sizes of state and action spaces of each agent. Finally, we design a Natural Policy Gradient (NPG) based algorithm that, in the three cases stated above, can converge to an optimal MARL policy within $\\mathcal{O}(e_j)$ error with a sample complexity of $\\mathcal{O}(e_j^{-3})$, $j\\in\\{1,2,3\\}$, respectively.

Motivation & Objective

• To address the lack of scalable, globally convergent methods for cooperative heterogeneous multi-agent reinforcement learning (MARL) with diverse agent types.
• To establish rigorous approximation guarantees between finite-population MARL and mean field control (MFC) for K heterogeneous agent classes.
• To develop a sample-efficient policy gradient algorithm that converges to a near-optimal MARL policy under the proposed MFC approximation.
• To analyze three distinct interaction regimes: joint distribution dependence, individual class distribution dependence, and marginal population distribution dependence.
• To generalize traditional MFC—assumed homogeneous agents—to heterogeneous systems with inter-class interactions and class-specific dynamics.

Proposed method

• Models a population of N_pop heterogeneous agents partitioned into K classes, each with N_k homogeneous agents, enabling class-level symmetry and inter-class interaction modeling.
• Derives three distinct MFC approximation scenarios based on how reward and transition functions depend on population distributions: joint, individual class, and marginal distributions.
• Establishes Lipschitz continuity of reward, transition, and policy functions in the MFC limit, enabling error propagation analysis via total variation distance.
• Translates the K-class MARL problem into an equivalent MFC problem using normalized class distributions μ̄ and ν̄, ensuring consistency across agent classes.
• Proves approximation error bounds using stability and contraction arguments, with explicit dependence on |X|, |U|, and 1/√N_k terms.
• Designs a Natural Policy Gradient (NPG) algorithm for the MFC problem, showing convergence within O(e_j) error and O(e_j^{-3}) sample complexity for each of the three scenarios.

Experimental results

Research questions

• RQ1 Can cooperative MARL with K heterogeneous agent classes be effectively approximated by a K-class mean field control (MFC) framework?
• RQ2 What is the approximation error between the finite-population MARL and its corresponding MFC limit under different dependency structures of rewards and transitions?
• RQ3 How does the error scale with the size of individual agent classes and the overall population in heterogeneous MARL systems?
• RQ4 Can a sample-efficient policy gradient algorithm be designed to converge to a near-optimal MARL policy via the MFC approximation?
• RQ5 What are the theoretical sample complexity and convergence rates of the proposed NPG-based algorithm across different interaction regimes?

Key findings

• The approximation error for the joint distribution scenario is bounded by e₁ = O((√|X| + √|U|)/N_pop × ∑ₖ √Nₖ), showing inverse dependence on class sizes.
• For the individual class distribution case, the error is e₂ = O((√|X| + √|U|) × ∑ₖ 1/√Nₖ), indicating stronger dependence on smaller classes.
• In the marginal distribution scenario, the error is e₃ = O((√|X| + √|U|) × (A/N_pop × ∑ₖ √Nₖ + B/√N_pop)), with additional constants A and B.
• The proposed NPG-based algorithm achieves convergence within O(e_j) error for each scenario j ∈ {1,2,3}, with sample complexity O(e_j^{-3}).
• Theoretical bounds are derived under Lipschitz continuity assumptions on reward, transition, and policy functions, ensuring stability in the MFC limit.
• The framework generalizes traditional MFC by allowing inter-class interactions and class-specific dynamics, enabling application to real-world heterogeneous systems like ride-sharing or smart grids.