[Paper Review] Episodic Curiosity through Reachability

Nikolay Savinov, Anton Raichuk+5 more|arXiv (Cornell University)|Oct 4, 2018

Reinforcement Learning in Robotics29 references162 citations

TL;DR

The paper introduces episodic curiosity via an episodic memory-based reachability predictor to provide a dense exploration bonus, improving performance in sparse-reward RL across 3D environments. It outperforms ICM on VizDoom/DMLab and enables first-person-view curiosity in MuJoCo.

ABSTRACT

Rewards are sparse in the real world and most of today's reinforcement learning algorithms struggle with such sparsity. One solution to this problem is to allow the agent to create rewards for itself - thus making rewards dense and more suitable for learning. In particular, inspired by curious behaviour in animals, observing something novel could be rewarded with a bonus. Such bonus is summed up with the real task reward - making it possible for RL algorithms to learn from the combined reward. We propose a new curiosity method which uses episodic memory to form the novelty bonus. To determine the bonus, the current observation is compared with the observations in memory. Crucially, the comparison is done based on how many environment steps it takes to reach the current observation from those in memory - which incorporates rich information about environment dynamics. This allows us to overcome the known "couch-potato" issues of prior work - when the agent finds a way to instantly gratify itself by exploiting actions which lead to hardly predictable consequences. We test our approach in visually rich 3D environments in ViZDoom, DMLab and MuJoCo. In navigational tasks from ViZDoom and DMLab, our agent outperforms the state-of-the-art curiosity method ICM. In MuJoCo, an ant equipped with our curiosity module learns locomotion out of the first-person-view curiosity only.

Motivation & Objective

• Address sparse-reward reinforcement learning by introducing an episodic curiosity module that produces a dense exploration bonus.
• Leverage episodic memory to compare current observations with past ones based on reachability (number of environment steps).
• Train a reachability network and an embedding/comparator pair to quantify novelty.
• Demonstrate robustness to couch-potato behaviors and improve exploration in VizDoom, DMLab, and MuJoCo tasks.

Proposed method

• Use a siamese embedding network E and a comparator C to form a reachability network R(o_i, o_j)=C(E(o_i), E(o_j)).
• Maintain an episodic memory M of past embeddings within an episode; store a current observation if novelty b exceeds a threshold.
• Compute a novelty bonus b from memory reachability via a function B(M, e) that depends on the estimated distance to memory items; augment task reward r with b.
• Train R-network using pairs of observations drawn from sequences; positives are temporally close within k, negatives are farther apart, using logistic regression loss.
• Integrate with PPO by adding the bonus to the task reward; optionally train R-network online or offline during policy learning.
• Compare against PPO baselines, PPO+ICM, and Grid Oracle across VizDoom, DMLab, and MuJoCo setups.

Experimental results

Research questions

• RQ1 Can reachability-based episodic memory provide a robust curiosity signal that avoids couch-potato behaviors seen in prediction-error based methods?
• RQ2 Does episodic curiosity improve learning efficiency and final performance in sparse-reward 3D environments compared to state-of-the-art baselines?
• RQ3 How does the approach perform under procedurally generated, highly variable levels and in no-reward exploration scenarios?
• RQ4 Is the curiosity signal compatible with dense-reward tasks without compromising performance?
• RQ5 Can the method generalize to first-person-view curiosity in continuous control domains (MuJoCo)?

Key findings

• EC outperforms the ICM baseline in VizDoom and DMLab navigation tasks.
• In procedurally generated DMLab levels, EC yields at least 2x higher success rates relative to ICM on sparse targets.
• Under no-reward exploration, EC covers substantially more area than ICM (no-reward/no-fire variants show ICM struggling).
• In dense-reward DMLab tasks, EC does not significantly degrade PPO performance compared to baselines.
• An ant in MuJoCo learns locomotion from first-person view using the EC reward signal alone.
• Across all benchmarks, EC provides faster convergence and more robust exploration than prior curiosity methods.