[Paper Review] Emergent Tool Use From Multi-Agent Autocurricula

Bowen Baker, Ingmar Kanitscheider+5 more|arXiv (Cornell University)|Sep 17, 2019

Reinforcement Learning in Robotics70 references335 citations

TL;DR

The paper shows that multi-agent self-play in a physics-based hide-and-seek environment induces a self-supervised autocurriculum with six emergent strategies, including tool use, and proposes transfer-based evaluation and targeted intelligence tests.

ABSTRACT

Through multi-agent competition, the simple objective of hide-and-seek, and standard reinforcement learning algorithms at scale, we find that agents create a self-supervised autocurriculum inducing multiple distinct rounds of emergent strategy, many of which require sophisticated tool use and coordination. We find clear evidence of six emergent phases in agent strategy in our environment, each of which creates a new pressure for the opposing team to adapt; for instance, agents learn to build multi-object shelters using moveable boxes which in turn leads to agents discovering that they can overcome obstacles using ramps. We further provide evidence that multi-agent competition may scale better with increasing environment complexity and leads to behavior that centers around far more human-relevant skills than other self-supervised reinforcement learning methods such as intrinsic motivation. Finally, we propose transfer and fine-tuning as a way to quantitatively evaluate targeted capabilities, and we compare hide-and-seek agents to both intrinsic motivation and random initialization baselines in a suite of domain-specific intelligence tests.

Motivation & Objective

• Motivate unsupervised skill discovery in open-ended, physically grounded environments.
• Demonstrate that multi-agent competition induces autocurricula with progressing strategies.
• Show emergence of human-relevant skills such as tool use and coordination.
• Propose transfer learning and targeted intelligence tests to evaluate open-ended agents.
• Open-source the environment and code to enable further research.

Proposed method

• Use a mixed competitive/cooperative physics-based hide-and-seek environment.
• Train agents with Proximal Policy Optimization (PPO) and Generalized Advantage Estimation (GAE) under decentralized execution and centralized training.
• Employ an ego-centric, entity-based attention policy architecture with self-attention over variable numbers of entities.
• Observe emergence of up to six strategy phases through self-play, driven solely by the hide-and-seek objective.
• Compare multi-agent autocurricula to intrinsic motivation baselines and random initialization in domain-specific tests.
• Propose transfer and fine-tuning as evaluation frameworks using a suite of intelligence tasks.

Experimental results

Research questions

• RQ1 Can multi-agent competition induce autocurricula that yield complex, tool-using behaviors in a physically grounded environment?
• RQ2 What are the emergent phases of strategy as agents train against each other?
• RQ3 Does multi-agent autocurriculum scale with environment complexity, and how does it compare to intrinsic motivation alone?
• RQ4 Can transfer learning and targeted intelligence tests quantify open-ended learning progress?
• RQ5 How do pretrained agents perform on domain-specific manipulation and cognition tasks relative to baselines?

Key findings

• Agents exhibit up to six distinct stages of strategy and counter-strategy during training.
• Hiders learn to build shelters from movable boxes and walls; seekers learn to use ramps to penetrate forts.
• Seeker and hider strategies include ramp use, ramp defense, box surfing, and surf defense.
• Multi-agent autocurricula scale with environment complexity and yield more human-relevant behaviors than intrinsic motivation baselines.
• Transfer experiments show improved or faster convergence on 3 of 5 targeted tasks for hide-and-seek pretrained agents compared to baselines.
• The work provides open-source environments and code to support further research.