[Paper Review] Open-Ended Learning Leads to Generally Capable Agents

Open Ended Learning Team, Adam Stooke+16 more|arXiv (Cornell University)|Jul 27, 2021

Reinforcement Learning in Robotics74 references55 citations

TL;DR

The paper presents XLand, a massively multi-task 3D environment and an open-ended training loop that yields agents with zero-shot generalization and broad competence across a diverse task space, outperforming fixed-distribution RL. It shows dynamic task generation and iterative generations with policy distillation drive continual learning and emergent behaviors.

ABSTRACT

In this work we create agents that can perform well beyond a single, individual task, that exhibit much wider generalisation of behaviour to a massive, rich space of challenges. We define a universe of tasks within an environment domain and demonstrate the ability to train agents that are generally capable across this vast space and beyond. The environment is natively multi-agent, spanning the continuum of competitive, cooperative, and independent games, which are situated within procedurally generated physical 3D worlds. The resulting space is exceptionally diverse in terms of the challenges posed to agents, and as such, even measuring the learning progress of an agent is an open research problem. We propose an iterative notion of improvement between successive generations of agents, rather than seeking to maximise a singular objective, allowing us to quantify progress despite tasks being incomparable in terms of achievable rewards. We show that through constructing an open-ended learning process, which dynamically changes the training task distributions and training objectives such that the agent never stops learning, we achieve consistent learning of new behaviours. The resulting agent is able to score reward in every one of our humanly solvable evaluation levels, with behaviour generalising to many held-out points in the universe of tasks. Examples of this zero-shot generalisation include good performance on Hide and Seek, Capture the Flag, and Tag. Through analysis and hand-authored probe tasks we characterise the behaviour of our agent, and find interesting emergent heuristic behaviours such as trial-and-error experimentation, simple tool use, option switching, and cooperation. Finally, we demonstrate that the general capabilities of this agent could unlock larger scale transfer of behaviour through cheap finetuning.

Motivation & Objective

• Motivate the creation of agents that generalize beyond a single task within a vast, procedurally generated environment.
• Define and study an environment space (XLand) that combines worlds, games, and co-player policies to create a massive, smoothly varying task space.
• Develop an open-ended training process that continually changes task distributions and objectives to sustain learning across percentiles of performance.
• Quantify progress using normalised score percentiles and analyze emergent general behaviors across held-out evaluation tasks.

Proposed method

• Introduce XLand: a natively multi-agent, procedurally generated 3D world space with controllable agents, objects, gadgets, and rewards.
• Represent tasks as worlds, games, and co-player policies, forming a vast task space that is smooth and rich in diversity.
• Train via deep RL with an attention-based network that implicitly models game goals, using dynamically generated training tasks.
• Use an iterative, population-based training regime with policy distillation across generations to bootstrap new policies and redefine performance frontiers.
• Measure progress with normalised score percentiles across the evaluation space and compare dynamic task generation to uniform sampling.

Experimental results

Research questions

• RQ1 Can agents trained in an open-ended, dynamically generated task space achieve zero-shot generalization across held-out evaluation tasks?
• RQ2 Does dynamically varying training tasks outperform fixed distributions in enabling continual learning across a vast space of tasks?
• RQ3 What kinds of emergent heuristics and multi-agent behaviors arise in generally capable agents trained with open-ended learning?
• RQ4 To what extent can finetuning enhance performance after zero-shot learning within this open-ended framework?

Key findings

• Agents exhibit zero-shot generalization across diverse evaluation levels, including Hide and Seek, Capture the Flag, and Tag.
• Finetuning on a new task for around 100 million steps can yield drastic performance gains compared to zero-shot or training from scratch.
• Emergent behaviors such as directed exploration, information gathering through other players, and cooperative dynamics appear in evaluation scenarios.
• Dynamic task generation is crucial for learning, outperforming uniform sampling from the task space.
• The general capabilities of the agent suggest potential for more scalable transfer of behavior through inexpensive finetuning.