[Paper Review] Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?

Christian Schroeder de Witt, Tarun Gupta+5 more|arXiv (Cornell University)|Nov 18, 2020

Reinforcement Learning in Robotics33 references182 citations

TL;DR

The paper shows that Independent PPO (IPPO), an independent learning approach, can match or outperform state‑of‑the‑art centralized training with decentralized execution methods on SMAC, with limited hyperparameter tuning. It also analyzes the role of policy clipping and central state information, suggesting relative overgeneralisation may be less problematic in SMAC than theory predicts.

ABSTRACT

Most recently developed approaches to cooperative multi-agent reinforcement learning in the \emph{centralized training with decentralized execution} setting involve estimating a centralized, joint value function. In this paper, we demonstrate that, despite its various theoretical shortcomings, Independent PPO (IPPO), a form of independent learning in which each agent simply estimates its local value function, can perform just as well as or better than state-of-the-art joint learning approaches on popular multi-agent benchmark suite SMAC with little hyperparameter tuning. We also compare IPPO to several variants; the results suggest that IPPO's strong performance may be due to its robustness to some forms of environment non-stationarity.

Motivation & Objective

• Motivate a reassessment of independent learning viability in cooperative MARL tasks like SMAC.
• Evaluate IPPO against centralized value function methods such as QMIX, MAVEN, and MAPPO on hard SMAC maps.
• Investigate why IPPO performs well, focusing on PPO clipping and the utility of centralised state information during training.

Proposed method

• Propose Independent PPO (IPPO) where each agent learns a local policy with clipping based on an independent objective.
• Use a local critic V_phi(z_t^a) shared across agents with Generalized Advantage Estimation (GAE) for each agent.
• Define the per-agent advantage A_t^a via TD errors and multi-step GAE as in equation (4).
• Apply policy clipping in the PPO objective and also apply optional value clipping to restrict critic updates as in equation (6).
• Train with shared network parameters across agents for critics and between actors, using a centralized training with decentralized execution setting.
• Evaluate on 16 SMAC maps with mild map-specific hyperparameter tuning and compare to QMIX, IQL, MAPPO, and MAVEN.

Experimental results

Research questions

• RQ1 Does IPPO match or exceed state-of-the-art CTDE MARL methods on SMAC across varied maps?
• RQ2 How do PPO-specific components like policy clipping and value clipping influence performance in independent learning for cooperative MARL?
• RQ3 What is the impact of conditioning critics on full state information during centralized training for IPPO?
• RQ4 Is relative overgeneralisation a practical obstacle for IPPO on SMAC maps?
• RQ5 How do IPPO results compare to independent baselines (IAC, IQL) and centralized baselines (QMIX, MAPPO, MAVEN) on hard SMAC maps?

Key findings

• IPPO significantly outperforms MAPPO and QMIX on several hard SMAC maps.
• IPPO beats IQL and IAC and shows greater stability across many maps.
• Policy clipping is essential to IPPO’s performance, while value clipping improves some maps.
• Using full central state information for the critic can be worse than local critics on hard maps, indicating central state information is not universally beneficial in SMAC.
• Reducing the effective learning rate via IPPO’s clipping cannot be replicated by simply lowering the learning rate in IAC, suggesting clipping provides a unique stabilizing benefit.