[Paper Review] Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning

Qian Long, Zihan Zhou+4 more|arXiv (Cornell University)|Mar 23, 2020

Reinforcement Learning in Robotics52 references38 citations

TL;DR

This paper introduces Evolutionary Population Curriculum (EPC), a curriculum learning framework that scales multi-agent reinforcement learning by progressively increasing agent population and using evolutionary selection to maintain adaptability across stages.

ABSTRACT

In multi-agent games, the complexity of the environment can grow exponentially as the number of agents increases, so it is particularly challenging to learn good policies when the agent population is large. In this paper, we introduce Evolutionary Population Curriculum (EPC), a curriculum learning paradigm that scales up Multi-Agent Reinforcement Learning (MARL) by progressively increasing the population of training agents in a stage-wise manner. Furthermore, EPC uses an evolutionary approach to fix an objective misalignment issue throughout the curriculum: agents successfully trained in an early stage with a small population are not necessarily the best candidates for adapting to later stages with scaled populations. Concretely, EPC maintains multiple sets of agents in each stage, performs mix-and-match and fine-tuning over these sets and promotes the sets of agents with the best adaptability to the next stage. We implement EPC on a popular MARL algorithm, MADDPG, and empirically show that our approach consistently outperforms baselines by a large margin as the number of agents grows exponentially.

Motivation & Objective

• Motivate the challenge of learning in environments with exponentially growing agent populations.
• Propose a population-invariant policy/critic architecture that generalizes to variable numbers of agents.
• Introduce an evolutionary selection mechanism to address objective misalignment across curriculum stages.
• Demonstrate EPC’s scalability and robustness by applying it to MADDPG across diverse multi-agent tasks.

Proposed method

• Adopt a self-attention based, population-invariant architecture for Q-functions and policies to handle arbitrary agent counts.
• Divide training into stages with increasing agent populations to form a curriculum.
• Maintain K parallel agent sets per role and perform mix-and-match (crossover) across sets to create scaled populations.
• Use MARL fine-tuning as a guided mutation operator during curriculum growth.
• Apply an evolutionary selection process to choose the best-adapting agent sets for the next stage, based on fitness across scaled environments.
• Demonstrate EPC on MADDPG and compare against baselines across three environments.

Experimental results

Research questions

• RQ1 How can agent population scaling be performed without losing stability or performance in MARL?
• RQ2 Does an evolutionary mix-and-match approach improve adaptation to larger populations compared to simple cloning?
• RQ3 Can attention-based, population-invariant architectures support scalable MARL training across arbitrary agent counts?
• RQ4 What gains does EPC provide over vanilla population curricula and non-curriculum MARL baselines as the number of agents grows exponentially?

Key findings

• EPC consistently outperforms baselines as the number of agents grows, including exponential increases in population.
• Attention-based, population-invariant architectures improve MADDPG performance compared to baseline MADDPG and mean-field methods.
• Vanilla population curricula degrade as population scales, while EPC maintains superior performance across scales.
• EPC yields higher survival and more grass consumption in Grassland, and better collaboration and resource collection in Adversarial Battle and Food Collection.