[Paper Review] Large-Scale Study of Curiosity-Driven Learning

Yuri Burda, Harri Edwards+4 more|arXiv (Cornell University)|Aug 13, 2018

Psychological and Educational Research StudiesPsychology41 references364 citations

TL;DR

The paper conducts a large-scale empirical study of learning driven solely by intrinsic curiosity without extrinsic rewards across 54 environments, comparing feature spaces for forward dynamics and highlighting both strengths and limitations of prediction-error based curiosity.

ABSTRACT

Reinforcement learning algorithms rely on carefully engineering environment rewards that are extrinsic to the agent. However, annotating each environment with hand-designed, dense rewards is not scalable, motivating the need for developing reward functions that are intrinsic to the agent. Curiosity is a type of intrinsic reward function which uses prediction error as reward signal. In this paper: (a) We perform the first large-scale study of purely curiosity-driven learning, i.e. without any extrinsic rewards, across 54 standard benchmark environments, including the Atari game suite. Our results show surprisingly good performance, and a high degree of alignment between the intrinsic curiosity objective and the hand-designed extrinsic rewards of many game environments. (b) We investigate the effect of using different feature spaces for computing prediction error and show that random features are sufficient for many popular RL game benchmarks, but learned features appear to generalize better (e.g. to novel game levels in Super Mario Bros.). (c) We demonstrate limitations of the prediction-based rewards in stochastic setups. Game-play videos and code are at https://pathak22.github.io/large-scale-curiosity/

Motivation & Objective

• Motivate intrinsic curiosity as a scalable alternative to hand-designed extrinsic rewards in reinforcement learning.
• Systematically study curiosity-driven learning across 54 environments including Atari, Mario, and 3D navigation.
• Evaluate how different feature spaces for forward dynamics affect curiosity-based exploration.
• Assess scalability, stability, and generalization of curiosity-driven agents without extrinsic rewards.

Proposed method

• Use a dynamics-based intrinsic reward defined as r_t = -log p(phi(x_{t+1}) | x_t, a_t) (surprisal) derived from forward dynamics.
• Compare feature spaces for embedding observations phi: raw pixels, random features, inverse dynamics features (IDF), and variational autoencoders (VAE).
• Employ PPO with reward and advantage normalization, observation normalization, many parallel actors, and feature batch normalization to stabilize training.
• Remove end-of-episode signals to study infinite-horizon, purely curiosity-driven exploration.
• Evaluate across 54 environments (Atari, Mario, Roboschool, Unity) and analyze generalization to novel levels.

Experimental results

Research questions

• RQ1 Can purely curiosity-driven agents learn meaningful behaviors without extrinsic rewards across diverse environments?
• RQ2 How do different observation embedding strategies (RF, VAE, IDF, Pixels) affect curiosity-driven exploration and generalization?
• RQ3 Does curiosity-based exploration align with extrinsic rewards in human-designed environments, and what are its limitations in stochastic settings?
• RQ4 To what extent can skills learned via curiosity transfer to novel levels or environments without additional rewards?

Key findings

• Curiosity-driven agents can acquire extrinsic rewards in many Atari games without any extrinsic training rewards.
• Random features often provide a simple and stable embedding for curiosity across many benchmarks; learned features generalize better to novel levels (e.g., Mario).
• Inverse dynamics features outperform random features in about 55% of Atari games, while raw pixels perform poorly for forward dynamics.
• In Mario, increasing batch size (from 128 to 2048 parallel threads) significantly improves exploration performance and level discovery.
• Curiosity can help in sparse-reward or terminal-reward tasks, improving performance where extrinsic-only training fails to progress.
• Stochasticity (noisy TV) can mislead curiosity, slowing learning but not permanently preventing eventual extrinsic reward attainment in some cases.