[Paper Review] Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions

Rui Wang, Joel Lehman+5 more|arXiv (Cornell University)|Mar 19, 2020

Reinforcement Learning in Robotics61 references43 citations

TL;DR

Enhanced POET extends POET with domain-general novelty measurement, efficient goal-switching, a CPPN-based environmental encoding, and a new open-endedness progress metric, demonstrating stronger open-ended reinforcement learning.

ABSTRACT

Creating open-ended algorithms, which generate their own never-ending stream of novel and appropriately challenging learning opportunities, could help to automate and accelerate progress in machine learning. A recent step in this direction is the Paired Open-Ended Trailblazer (POET), an algorithm that generates and solves its own challenges, and allows solutions to goal-switch between challenges to avoid local optima. However, the original POET was unable to demonstrate its full creative potential because of limitations of the algorithm itself and because of external issues including a limited problem space and lack of a universal progress measure. Importantly, both limitations pose impediments not only for POET, but for the pursuit of open-endedness in general. Here we introduce and empirically validate two new innovations to the original algorithm, as well as two external innovations designed to help elucidate its full potential. Together, these four advances enable the most open-ended algorithmic demonstration to date. The algorithmic innovations are (1) a domain-general measure of how meaningfully novel new challenges are, enabling the system to potentially create and solve interesting challenges endlessly, and (2) an efficient heuristic for determining when agents should goal-switch from one problem to another (helping open-ended search better scale). Outside the algorithm itself, to enable a more definitive demonstration of open-endedness, we introduce (3) a novel, more flexible way to encode environmental challenges, and (4) a generic measure of the extent to which a system continues to exhibit open-ended innovation. Enhanced POET produces a diverse range of sophisticated behaviors that solve a wide range of environmental challenges, many of which cannot be solved through other means.

Motivation & Objective

• Advance open-ended reinforcement learning by creating and solving an unbounded sequence of learning challenges.
• Develop a domain-general novelty metric to guide environment invention beyond hand-crafted encodings.
• Improve the efficiency of transferring solutions between environments to sustain scalable open-ended search.
• Introduce a more expressive environmental encoding to generate diverse, intricate environments.
• Propose and validate a quantitative measure of open-ended progress (ANNECS) to track continual innovation.

Proposed method

• Introduce PATA-EC (Performance of All Transferred Agents Environment Characterization) to quantify novelty based on how all agents perform across an environment.
• Replace domain-specific novelty with a domain-general distance metric; normalize and use Euclidean distance for novelty assessments.
• Improve the transfer mechanism by requiring the maximum of the last five incumbent scores to be exceeded for a transfer, reducing noise and computation.
• Adopt a CPPN-based environmental encoding (via NEAT) to generate more complex and diverse landscapes beyond hand-crafted obstacles.
• Use ANNECS (Accumulated Number of Novel Environments Created and Solved) to measure ongoing open-ended progress across a run.
• Demonstrate Enhanced POET in a 2-D bipedal walking domain with obstacle courses encoded by CPPNs, evaluating diversity, transfer efficiency, and open-endedness metrics.

Experimental results

Research questions

• RQ1 Can a domain-general novelty measure (PATA-EC) effectively guide the creation of meaningful and diverse environments across domains?
• RQ2 Does an improved transfer mechanism reduce computation while maintaining or improving solution discovery?
• RQ3 Does a CPPN-based environmental encoding enable richer, more diverse open-ended environments than hand-crafted encodings?
• RQ4 Does the ANNECS metric reliably reflect ongoing open-ended progress in Enhanced POET?
• RQ5 To what extent can Enhanced POET demonstrate open-ended learning beyond original POET in a more expressive domain?

Key findings

• PATA-EC enables domain-general environmental novelty, achieving diversity comparable to hand-crafted novelty with about a 82% increase in computation in ES steps in the tested domain.
• An improved transfer strategy reduces computation to about 79.7% of original POET’s cost while preserving diversity and problem-solving capabilities.
• CPPN-based environmental encoding yields a wide variety of obstacle configurations, producing qualitatively richer environments than hand-crafted encodings.
• Enhanced POET with CPPN-based encoding shows deep, hierarchically nested environmental phylogenies, indicating open-ended exploration.
• Control experiments show POET-solvable environments require self-generated curricula; direct optimization and manual curricula underperform POET’s implicit curriculum.
• ANNECS increases over time, indicating continual generation and solving of novel environments across runs.