[Paper Review] Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions

Rui Wang, Joel Lehman+2 more|arXiv (Cornell University)|Jan 7, 2019

Reinforcement Learning in Robotics69 references125 citations

TL;DR

POET pairs environmental challenge generation with agent optimization, enabling transfers of solutions across environments to produce diverse, increasingly complex learning curricula in a single run.

ABSTRACT

While the history of machine learning so far largely encompasses a series of problems posed by researchers and algorithms that learn their solutions, an important question is whether the problems themselves can be generated by the algorithm at the same time as they are being solved. Such a process would in effect build its own diverse and expanding curricula, and the solutions to problems at various stages would become stepping stones towards solving even more challenging problems later in the process. The Paired Open-Ended Trailblazer (POET) algorithm introduced in this paper does just that: it pairs the generation of environmental challenges and the optimization of agents to solve those challenges. It simultaneously explores many different paths through the space of possible problems and solutions and, critically, allows these stepping-stone solutions to transfer between problems if better, catalyzing innovation. The term open-ended signifies the intriguing potential for algorithms like POET to continue to create novel and increasingly complex capabilities without bound. Our results show that POET produces a diverse range of sophisticated behaviors that solve a wide range of environmental challenges, many of which cannot be solved by direct optimization alone, or even through a direct-path curriculum-building control algorithm introduced to highlight the critical role of open-endedness in solving ambitious challenges. The ability to transfer solutions from one environment to another proves essential to unlocking the full potential of the system as a whole, demonstrating the unpredictable nature of fortuitous stepping stones. We hope that POET will inspire a new push towards open-ended discovery across many domains, where algorithms like POET can blaze a trail through their interesting possible manifestations and solutions.

Motivation & Objective

• Motivate open-ended, self-generated curricula where problems and solutions co-evolve.
• Develop an algorithm that simultaneously grows environment complexity and optimizes agent policies.
• Enable transfer of solving strategies across environments to catalyze innovation.
• Demonstrate open-ended progress in a single run within a 2-D bipedal-walker domain.

Proposed method

• Maintain a population of environment–agent pairs (EA_List) beginning with a simple pair.
• Generate new environments by mutating environment encodings while ensuring they are neither too hard nor too easy for current agents and prioritizing novelty.
• Optimize each agent within its paired environment using Evolution Strategies (ES).
• Periodically attempt transfers of agent policies between environments to share useful skills and accelerate progress.
• Transfer attempts are evaluated and accepted if they improve performance in the target environment.
• Operate in parallel to leverage multiple processors and enable large-scale exploration.

Experimental results

Research questions

• RQ1 Can POET produce an open-ended sequence of increasingly complex and diverse environments within a single run?
• RQ2 Are transfers of solutions between environments essential for progress and innovation in POET?
• RQ3 Does POET achieve diverse, solvable challenges that could not be solved by direct optimization or a fixed curriculum?
• RQ4 How does the performance of agents evolved in POET compare to optimization on isolated environments?

Key findings

• POET generates a diverse set of challenging environments that are invented and solved within one run.
• Solutions to challenging environments could not be found by optimizing directly on those environments alone.
• Curriculum-based gradual scaling to the same challenges did not reach POET’s results; open-ended growth relies on environment diversity and transfer.
• Periodic transfers between environments are important for unlocking progress and enabling serendipitous stepping stones.
• A single run yields a broad range of sophisticated locomotion strategies across varied terrains.
• The transfer mechanism supports cross-pollination that accelerates advancement beyond individual environments.