[Paper Review] Go-Explore: a New Approach for Hard-Exploration Problems

Adrien Ecoffet, Joost Huizinga+3 more|arXiv (Cornell University)|Jan 30, 2019

Reinforcement Learning in Robotics99 references225 citations

TL;DR

Go-Explore introduces a two-phase algorithm that remembers promising states, returns to them without exploration, then explores from there, and finally robustifies with imitation learning, achieving superhuman Atari performance on hard-exploration tasks.

ABSTRACT

A grand challenge in reinforcement learning is intelligent exploration, especially when rewards are sparse or deceptive. Two Atari games serve as benchmarks for such hard-exploration domains: Montezuma's Revenge and Pitfall. On both games, current RL algorithms perform poorly, even those with intrinsic motivation, which is the dominant method to improve performance on hard-exploration domains. To address this shortfall, we introduce a new algorithm called Go-Explore. It exploits the following principles: (1) remember previously visited states, (2) first return to a promising state (without exploration), then explore from it, and (3) solve simulated environments through any available means (including by introducing determinism), then robustify via imitation learning. The combined effect of these principles is a dramatic performance improvement on hard-exploration problems. On Montezuma's Revenge, Go-Explore scores a mean of over 43k points, almost 4 times the previous state of the art. Go-Explore can also harness human-provided domain knowledge and, when augmented with it, scores a mean of over 650k points on Montezuma's Revenge. Its max performance of nearly 18 million surpasses the human world record, meeting even the strictest definition of "superhuman" performance. On Pitfall, Go-Explore with domain knowledge is the first algorithm to score above zero. Its mean score of almost 60k points exceeds expert human performance. Because Go-Explore produces high-performing demonstrations automatically and cheaply, it also outperforms imitation learning work where humans provide solution demonstrations. Go-Explore opens up many new research directions into improving it and weaving its insights into current RL algorithms. It may also enable progress on previously unsolvable hard-exploration problems in many domains, especially those that harness a simulator during training (e.g. robotics).

Motivation & Objective

• Address the challenge of exploration in sparse/deceptive reward settings.
• Improve performance on hard-exploration Atari benchmarks without heavy reliance on intrinsic motivation.
• Develop a two-phase framework: phase 1 for exploration with an archive of promising states, phase 2 for robustification via imitation learning.

Proposed method

• Store an archive of promising states (cells) and how to reach them.
• At each step, select a cell, return to it deterministically, then explore from it with stochastic actions.
• Update the archive when new cells are discovered or better trajectories appear, including trajectory, state, score, and length.
• Use two representations for cells: a simple no-domain-knowledge downsampled grayscale 11x8 image, and a domain-knowledge enriched representation (e.g., agent position, room, keys).
• Phase 2 uses imitation learning (Backward Algorithm) to robustify the phase-1 trajectories by training from near the end of the trajectory and gradually moving toward the start with PPO until achieving or surpassing the original score.
• The method distinguishes deterministic training ( Phase 1) from stochastic evaluation, enabling deterministic resets and then adding stochasticity during Phase 2 for robustness.

Experimental results

Research questions

• RQ1 Can an explicit archive of previously visited states and retracing to them before exploration improve performance in hard-exploration tasks?
• RQ2 Does combining deterministic exploration with subsequent robustification via imitation learning yield scalable improvements on sparse/deceptive reward settings?
• RQ3 To what extent does domain knowledge in cell representations accelerate discovery and performance in hard-exploration benchmarks?
• RQ4 How does Go-Explore address detachment and derailment issues that afflict traditional intrinsic-motivation approaches?
• RQ5 What are the performance gains on Montezuma’s Revenge and Pitfall with and without domain knowledge?

Key findings

• On Montezuma’s Revenge, without domain knowledge, Go-Explore scores over 43,000 points (nearly 4x previous state of the art).
• With easy-to-provide domain knowledge, Go-Explore achieves a mean of over 650,000 points and a max above 18 million, surpassing human world records by a large margin.
• On Pitfall, Go-Explore with domain knowledge achieves a mean of about 59,494 points and a max of 107,363, near the game’s possible maximum.
• Without domain knowledge, Montezuma’s Revenge mean score is 43,763, still dramatically higher than prior work.
• Go-Explore can automatically generate high-performing demonstrations suitable for imitation learning, outperforming prior imitation-learning results that relied on human demonstrations.