[Paper Review] Emergent Complexity via Multi-Agent Competition

Trapit Bansal, Jakub Pachocki+3 more|arXiv (Cornell University)|Oct 10, 2017

Reinforcement Learning in RoboticsComputer Science31 references146 citations

TL;DR

The paper demonstrates that self-play in competitive multi-agent environments can produce highly complex behaviors in simple 3D physical tasks, via a distributed PPO training framework with an exploration curriculum and opponent sampling strategies.

ABSTRACT

Reinforcement learning algorithms can train agents that solve problems in complex, interesting environments. Normally, the complexity of the trained agent is closely related to the complexity of the environment. This suggests that a highly capable agent requires a complex environment for training. In this paper, we point out that a competitive multi-agent environment trained with self-play can produce behaviors that are far more complex than the environment itself. We also point out that such environments come with a natural curriculum, because for any skill level, an environment full of agents of this level will have the right level of difficulty. This work introduces several competitive multi-agent environments where agents compete in a 3D world with simulated physics. The trained agents learn a wide variety of complex and interesting skills, even though the environment themselves are relatively simple. The skills include behaviors such as running, blocking, ducking, tackling, fooling opponents, kicking, and defending using both arms and legs. A highlight of the learned behaviors can be found here: https://goo.gl/eR7fbX

Motivation & Objective

• Motivate why competitive multi-agent self-play can generate complex behaviors beyond the environment itself.
• Introduce four competitive 3D environments with simple rules and physics.
• Show that a natural curriculum arises from training against opponents of similar skill levels.
• Demonstrate that an exploration curriculum facilitates learning under sparse rewards.

Proposed method

• Use Proximal Policy Optimization (PPO) in a decentralized, distributed training setup with large-scale rollouts across multiple GPUs.
• Employ two 3D agent bodies (ant and humanoid) in four tasks: Run to Goal, You Shall Not Pass, Sumo, Kick and Defend.
• Incorporate an exploration curriculum by annealing a dense exploration reward to zero over training.
• Adopt random old opponent sampling to stabilize self-play training and avoid rapid imbalance.
• Use GAE with a clipped PPO objective and train separate policies for asymmetric games when needed.

Experimental results

Research questions

• RQ1 Can self-play in competitive multi-agent environments yield emergent, complex behaviors beyond the environment's inherent complexity?
• RQ2 Does an exploration curriculum improve learning efficiency and enable discovery of non-trivial motor skills under sparse rewards?
• RQ3 What training strategies (e.g., opponent sampling, curriculum in randomization) yield robust policies in competitive 3D tasks?
• RQ4 How do the learned policies transfer to non-episodic or perturbed conditions (robustness tests)?

Key findings

• Competitive multi-agent training yields diverse emergent skills such as blocking, dodging, tackling, kicking, and defending.
• An exploration curriculum is crucial for learning under sparse rewards and increases sample efficiency.
• Training against randomly sampled older opponents stabilizes learning and fosters continual improvement.
• Ensembling multiple policies can improve robustness, especially for humanoid agents, compared to single-policy self-play.
• Randomization curricula in environment parameters help generalize policies without sacrificing early learning progress.