[Paper Review] Exploration by Random Network Distillation

Yuri Burda, Harrison Edwards+2 more|arXiv (Cornell University)|Oct 30, 2018

Reinforcement Learning in RoboticsComputer Science41 references257 citations

TL;DR

The paper introduces a simple intrinsic exploration bonus called random network distillation (RND) and shows it enables strong exploration in hard Atari games, achieving state-of-the-art on Montezuma’s Revenge without demonstrations. It also presents a method to combine intrinsic and extrinsic rewards for improved performance.

ABSTRACT

We introduce an exploration bonus for deep reinforcement learning methods that is easy to implement and adds minimal overhead to the computation performed. The bonus is the error of a neural network predicting features of the observations given by a fixed randomly initialized neural network. We also introduce a method to flexibly combine intrinsic and extrinsic rewards. We find that the random network distillation (RND) bonus combined with this increased flexibility enables significant progress on several hard exploration Atari games. In particular we establish state of the art performance on Montezuma's Revenge, a game famously difficult for deep reinforcement learning methods. To the best of our knowledge, this is the first method that achieves better than average human performance on this game without using demonstrations or having access to the underlying state of the game, and occasionally completes the first level.

Motivation & Objective

• Motivate exploration in deep RL when extrinsic rewards are sparse or hard to obtain.
• Propose a simple, scalable intrinsic reward based on predicting a fixed random network's output.
• Enable flexible integration of intrinsic and extrinsic rewards in policy optimization.
• Demonstrate effectiveness on hard exploration Atari games, including Montezuma’s Revenge, Gravitar, and Venture.

Proposed method

• Define a fixed randomly initialized target network that maps observations to a k-dimensional embedding f(x).
• Train a predictor network to minimize mean squared error to the target embedding: E[||f_hat(x; θ_hat) - f(x)||^2].
• Use the predictor’s prediction error as the intrinsic reward i_t added to extrinsic reward e_t: r_t = e_t + i_t.
• Normalize intrinsic rewards by a running standard deviation and whiten observations to stabilize learning.
• Combine intrinsic and extrinsic rewards with two value heads (V_E and V_I) to handle different reward streams and discounts.
• Scale experiments with large numbers of parallel environments to assess performance.
• Compare RND to dynamics-based exploration and PPO baselines to establish effectiveness on multiple hard exploration games.

Experimental results

Research questions

• RQ1 Does the RND-based exploration bonus improve exploration in environments with sparse rewards without requiring demonstrations or access to internal game state?
• RQ2 How does RND perform when combined with extrinsic rewards, and is there a benefit to using separate value heads for intrinsic and extrinsic rewards?
• RQ3 What are the effects of different discount factors on intrinsic vs extrinsic rewards for exploration and performance?
• RQ4 Can scaling the amount of parallel experience improve performance and final outcomes for RND-based agents?
• RQ5 How does RND compare to forward-dynamics exploration methods and baselines across multiple hard exploration Atari games?

Key findings

• RND achieves strong exploration on several hard Atari games, surpassing baselines and reaching state-of-the-art on Montezuma’s Revenge without demonstrations.
• Combining non-episodic intrinsic rewards with episodic extrinsic rewards (using two value heads) improves exploration over episodic-only setups in several experiments.
• Higher extrinsic discount factors improve performance, while higher intrinsic discount factors can hurt exploration.
• Larger-scale training with more parallel environments yields higher mean returns and better final performance, especially with recurrent policies.
• RND outperforms forward-dynamics-based exploration on Montezuma’s Revenge, Private Eye, and Solaris, and matches or exceeds PPO in several games.