[Paper Review] The Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent Games

Chao Yu, Akash Velu+4 more|arXiv (Cornell University)|Mar 2, 2021

Reinforcement Learning in RoboticsComputer Science32 references109 citations

TL;DR

This paper demonstrates that Multi-Agent Proximal Policy Optimization (MAPPO), an on-policy reinforcement learning algorithm, achieves strong performance in cooperative multi-agent environments—matching or exceeding off-policy baselines in sample efficiency and final performance—despite common assumptions about on-policy methods being less sample-efficient in multi-agent settings. The results are achieved with minimal hyperparameter tuning and no domain-specific modifications on a single GPU.

ABSTRACT

Proximal Policy Optimization (PPO) is a popular on-policy reinforcement learning algorithm but is significantly less utilized than off-policy learning algorithms in multi-agent settings. This is often due the belief that on-policy methods are significantly less sample efficient than their off-policy counterparts in multi-agent problems. In this work, we investigate Multi-Agent PPO (MAPPO), a variant of PPO which is specialized for multi-agent settings. Using a 1-GPU desktop, we show that MAPPO achieves surprisingly strong performance in three popular multi-agent testbeds: the particle-world environments, the Starcraft multi-agent challenge, and the Hanabi challenge, with minimal hyperparameter tuning and without any domain-specific algorithmic modifications or architectures. In the majority of environments, we find that compared to off-policy baselines, MAPPO achieves strong results while exhibiting comparable sample efficiency. Finally, through ablation studies, we present the implementation and algorithmic factors which are most influential to MAPPO's practical performance.

Motivation & Objective

• To challenge the prevailing belief that on-policy methods like PPO are less sample-efficient than off-policy methods in multi-agent reinforcement learning.
• To evaluate the practical effectiveness of Multi-Agent PPO (MAPPO) across diverse cooperative multi-agent environments.
• To identify key implementation and algorithmic factors that contribute to MAPPO’s strong performance in multi-agent settings.
• To assess whether MAPPO can achieve competitive results without domain-specific architectural or algorithmic modifications.

Proposed method

• MAPPO applies the Proximal Policy Optimization (PPO) algorithm to multi-agent environments by independently updating each agent’s policy using its own experience while maintaining on-policy learning constraints.
• The method uses a clipped probability ratio objective to stabilize training, ensuring policy updates remain within a trust region to prevent performance collapse.
• Each agent’s policy is optimized using a centralized critic that observes all agents’ observations and actions, enabling credit assignment in cooperative settings.
• The algorithm is trained on a single GPU, relying on standard neural network architectures without task-specific modifications.
• Hyperparameters are kept minimal and consistent across environments, with no environment-specific tuning.
• Ablation studies are conducted to isolate the impact of key components such as value function normalization, reward shaping, and policy update frequency.

Experimental results

Research questions

• RQ1 Does on-policy learning with MAPPO achieve competitive sample efficiency compared to off-policy baselines in cooperative multi-agent environments?
• RQ2 What are the key implementation factors that contribute to MAPPO’s strong empirical performance in multi-agent settings?
• RQ3 Can MAPPO achieve strong results without architectural or algorithmic modifications tailored to specific environments?
• RQ4 How does MAPPO perform across diverse multi-agent benchmarks such as particle-world, StarCraft, and Hanabi?

Key findings

• MAPPO achieves performance comparable to or better than off-policy baselines in the majority of environments tested, including particle-world, StarCraft II, and Hanabi.
• The algorithm demonstrates comparable sample efficiency to off-policy methods, challenging the assumption that on-policy methods are inherently less sample-efficient in multi-agent settings.
• Value function normalization and proper reward scaling are identified as critical implementation factors that significantly improve training stability and final performance.
• Minimal hyperparameter tuning is sufficient for strong performance across diverse environments, indicating robustness and generalization of the approach.
• Ablation studies show that policy update frequency and training stability are strongly influenced by the choice of normalization and clipping mechanisms.
• No domain-specific architectural changes or algorithmic modifications are required for MAPPO to achieve strong results, highlighting its simplicity and practicality.