[Paper Review] Major-Minor Mean Field Multi-Agent Reinforcement Learning

Kai Cui, Christian Fabián+2 more|arXiv (Cornell University)|Mar 19, 2023

Auction Theory and ApplicationsDecision Sciences3 citations

TL;DR

This paper introduces Major-Minor Mean Field Control (M3FC), a novel framework for cooperative multi-agent reinforcement learning that models systems with both many stochastic minor agents and one or more complex major agents. It establishes rigorous approximation properties and a dynamic programming principle, and proposes the M3FPPO algorithm, which successfully learns effective policies in diverse M3FC-type environments, demonstrating transferability across system sizes.

ABSTRACT

Multi-agent reinforcement learning (MARL) remains difficult to scale to many agents. Recent MARL using Mean Field Control (MFC) provides a tractable and rigorous approach to otherwise difficult cooperative MARL. However, the strict MFC assumption of many independent, weakly-interacting agents is too inflexible in practice. We generalize MFC to instead simultaneously model many similar and few complex agents -- as Major-Minor Mean Field Control (M3FC). Theoretically, we give approximation results for finite agent control, and verify the sufficiency of stationary policies for optimality together with a dynamic programming principle. Algorithmically, we propose Major-Minor Mean Field MARL (M3FMARL) for finite agent systems instead of the limiting system. The algorithm is shown to approximate the policy gradient of the underlying M3FC MDP. Finally, we demonstrate its capabilities experimentally in various scenarios. We observe a strong performance in comparison to state-of-the-art policy gradient MARL methods.

Motivation & Objective

• To address the limitation of existing mean field control (MFC) methods, which assume only homogeneous, minor agents and cannot model complex, influential major agents.
• To develop a theoretically grounded framework for cooperative multi-agent reinforcement learning that incorporates both major agents and stochastic, correlated minor agents.
• To establish approximation properties and a dynamic programming principle for M3FC in finite and infinite-horizon settings.
• To design and evaluate a new MARL algorithm, M3FPPO, capable of learning effective policies in M3FC-type problems.
• To demonstrate the transferability of policies trained on small systems to larger systems, enabling scalable deployment.

Proposed method

• Proposes a discrete-time generalization of Markov decision processes and MFC to model systems with both major agents and many minor agents, allowing for stochastic minor agent distributions influenced by the major agent state.
• Introduces a major-minor mean field control (M3FC) framework where minor agents' behavior depends on the major agent's state and the empirical distribution of minor agents, enabling strong correlations.
• Derives a dynamic programming principle for solving M3FC problems, enabling recursive optimization of policies.
• Establishes existence of an optimal stationary policy in the infinite-horizon discounted case, extending theoretical foundations to cooperative MFC.
• Proposes the Major-Minor Mean Field Proximal Policy Optimization (M3FPPO) algorithm, a novel MARL algorithm tailored for M3FC problems.
• Employs a centralized training with decentralized execution (CTDE) paradigm, where the major agent observes the full system state, while minor agents act based on local observations and the major agent’s state.

Experimental results

Research questions

• RQ1 Can a mean field control framework be extended to include both major agents and stochastic, correlated minor agents, beyond the standard i.i.d. minor agent assumption?
• RQ2 What are the theoretical approximation properties of M3FC in finite multi-agent systems, and how do they compare to existing MFC models?
• RQ3 Does the dynamic programming principle hold for M3FC, and can it be used to derive optimal policies?
• RQ4 Can the M3FPPO algorithm effectively learn cooperative policies in M3FC-type environments, especially when trained on small systems and transferred to larger ones?
• RQ5 How does M3FPPO compare to standard MARL baselines like IPPO in terms of performance and generalization across system sizes?

Key findings

• M3FPPO successfully learns effective policies in all tested M3FC environments, including 2G, Formation, Beach Bar, Foraging, and Potential, with strong performance even when trained on small finite systems.
• In the Foraging and Beach Bar environments, M3FPPO outperforms IPPO when both are trained directly on finite systems, demonstrating superior sample efficiency and policy quality.
• Training M3FPPO on small systems (N=5,10,20) and transferring the policy to larger systems yields comparable performance, indicating strong generalization and transferability.
• Theoretical analysis confirms the existence of an optimal stationary policy in the infinite-horizon discounted M3FC setting, extending the theoretical validity of the framework.
• The dynamic programming principle for M3FC is rigorously derived, enabling recursive policy optimization and providing a foundation for scalable solution methods.
• The framework successfully models complex dependencies where minor agents are correlated through the major agent’s state, enabling modeling of arbitrary system-level effects not tied to individual agents.