[Paper Review] Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning

Joshua Achiam, S. Shankar Sastry|arXiv (Cornell University)|Mar 6, 2017

Reinforcement Learning in RoboticsComputer Science14 references100 citations

TL;DR

The paper introduces surprise-based intrinsic rewards, via a learned transition model, to drive exploration in deep reinforcement learning, using surprisal and k-step learning progress as incentives, showing improved exploration across continuous control and Atari RAM tasks.

ABSTRACT

Exploration in complex domains is a key challenge in reinforcement learning, especially for tasks with very sparse rewards. Recent successes in deep reinforcement learning have been achieved mostly using simple heuristic exploration strategies such as $ε$-greedy action selection or Gaussian control noise, but there are many tasks where these methods are insufficient to make any learning progress. Here, we consider more complex heuristics: efficient and scalable exploration strategies that maximize a notion of an agent's surprise about its experiences via intrinsic motivation. We propose to learn a model of the MDP transition probabilities concurrently with the policy, and to form intrinsic rewards that approximate the KL-divergence of the true transition probabilities from the learned model. One of our approximations results in using surprisal as intrinsic motivation, while the other gives the $k$-step learning progress. We show that our incentives enable agents to succeed in a wide range of environments with high-dimensional state spaces and very sparse rewards, including continuous control tasks and games in the Atari RAM domain, outperforming several other heuristic exploration techniques.

Motivation & Objective

• Motivate exploration in deep RL for environments with sparse rewards.
• Develop scalable intrinsic rewards based on the mismatch between true and learned transition dynamics.
• Learn a transition model concurrently with the policy to guide exploration.
• Compare surprisal and k-step learning progress incentives against existing exploration methods including VIME.

Proposed method

• Formulate intrinsic reward as the KL-divergence between true P and learned P_phi and derive two scalable approximations.
• Use surprisal: intrinsic reward proportional to -log P_phi(s'|s,a).
• Use k-step learning progress: intrinsic reward based on log P_phi_t(s'|s,a) - log P_phi_{t-k}(s'|s,a).
• jointly update a dynamics model P_phi via supervised-like loss with regularization and KL-divergence constraint (Eq. 11).
• Update policy to maximize environment return plus eta times expected KL between true and learned dynamics (Eq. 2).
• Adjust eta to keep intrinsic rewards bounded and normalize bonuses for stability.

Experimental results

Research questions

• RQ1 Do surprisal and learning progress as intrinsic rewards improve exploration in high-dimensional, sparse-reward DRL settings?
• RQ2 How do these intrinsic incentives compare to existing methods like VIME and L2 model prediction error in continuous control and Atari RAM domains?
• RQ3 Can a single forward dynamics model provide scalable, robust intrinsic motivation across diverse tasks including deterministic and stochastic dynamics?
• RQ4 What is the impact of k on learning progress bonuses across tasks?

Key findings

• Surprisal incentives yield robust, improved exploration across a wide range of tasks including continuous control and Atari RAM domains.
• k-step learning progress can help in some tasks but may underperform surprisal depending on environment and k value.
• Surprisal often outperforms L2 model prediction error and competes with VIME at lower computational cost.
• The approach works with a fully-factored Gaussian dynamics model and forward passes, providing speedups over VIME.
• Surprisal generally outperforms other intrinsic motivation baselines in harder tasks like SwimmerGather and Venture-RAM.
• Surprisal remains effective even when naive exploration would fail, indicating it drives meaningful exploration in sparse-reward settings.