[Paper Review] Unifying Count-Based Exploration and Intrinsic Motivation

Marc G. Bellemare, Sriram Srinivasan+4 more|arXiv (Cornell University)|Jun 6, 2016

Reinforcement Learning in Robotics50 references246 citations

TL;DR

The paper introduces pseudo-counts derived from density models to generalize count-based exploration to non-tabular settings and connects them to information gain, showing improved exploration in Atari 2600 games including Montezuma’s Revenge.

ABSTRACT

We consider an agent's uncertainty about its environment and the problem of generalizing this uncertainty across observations. Specifically, we focus on the problem of exploration in non-tabular reinforcement learning. Drawing inspiration from the intrinsic motivation literature, we use density models to measure uncertainty, and propose a novel algorithm for deriving a pseudo-count from an arbitrary density model. This technique enables us to generalize count-based exploration algorithms to the non-tabular case. We apply our ideas to Atari 2600 games, providing sensible pseudo-counts from raw pixels. We transform these pseudo-counts into intrinsic rewards and obtain significantly improved exploration in a number of hard games, including the infamously difficult Montezuma's Revenge.

Motivation & Objective

• Motivate the problem of exploration in non-tabular reinforcement learning and the limitations of traditional count-based methods.
• Propose a density-model based mechanism to derive pseudo-counts that generalize counts across states.
• Establish theoretical connections between pseudo-counts, prediction gain, and information gain.
• Demonstrate the practical effectiveness of pseudo-count bonuses on Atari 2600 games, including Montezuma’s Revenge, and in actor-critic and replay settings.

Proposed method

• Define a pseudo-count from a density model by relating the model's current and recoded probabilities via rho_n and rho'_n.
• Use the recoding probability to derive a pseudo-count N_hat_n(x) that generalizes empirical counts N_n(x) to non-tabular spaces.
• Relate pseudo-counts to information gain and prediction gain, proving IG_n(x) ≤ PG_n(x) ≤ N_hat_n(x)^{-1}, and PG_n(x) ≤ N_hat_n(x)^{-1/2}.
• Apply the pseudo-count-based exploration bonus R^+_n(x,a) = β (N_hat_n(x) + 0.01)^{-1/2} within MBIE-EB style planning and in DQN/A3C frameworks.
• Validate properties of pseudo-counts on a simple Atari example (Freeway) and extend experiments to Atari 2600 games using CTS density model for pixels.

Experimental results

Research questions

• RQ1 Can a density-model derived pseudo-count generalize visit counts to non-tabular state spaces?
• RQ2 How do pseudo-counts relate to information gain and prediction gain, and can they provide theoretical guarantees for exploration?
• RQ3 Do pseudo-count based bonuses improve exploration in hard Atari games, including Montezuma’s Revenge, in both value-based and policy-based RL methods?

Key findings

• Pseudo-counts provide a meaningful, generalizable notion of state novelty in non-tabular settings.
• Prediction gain approximates information gain and bounds the exploration bonus via a relation with pseudo-counts.
• Pseudo-count bonuses enable significantly improved exploration in hard Atari games, notably Montezuma’s Revenge, compared to baselines.
• Incorporating pseudo-count bonuses with A3C (A3C+) yields better median performance across 60 Atari games than A3C alone.
• The CTS-based pseudo-counts enable faster exploration and higher scores in Montezuma’s Revenge within given frame budgets.