[Paper Review] #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning

Haoran Tang, Rein Houthooft+7 more|arXiv (Cornell University)|Nov 15, 2016

Reinforcement Learning in Robotics35 references343 citations

TL;DR

The paper extends classic count-based exploration to high-dimensional spaces via hashing (static and learned hashes) and shows near state-of-the-art results on continuous control and Atari benchmarks.

ABSTRACT

Count-based exploration algorithms are known to perform near-optimally when used in conjunction with tabular reinforcement learning (RL) methods for solving small discrete Markov decision processes (MDPs). It is generally thought that count-based methods cannot be applied in high-dimensional state spaces, since most states will only occur once. Recent deep RL exploration strategies are able to deal with high-dimensional continuous state spaces through complex heuristics, often relying on optimism in the face of uncertainty or intrinsic motivation. In this work, we describe a surprising finding: a simple generalization of the classic count-based approach can reach near state-of-the-art performance on various high-dimensional and/or continuous deep RL benchmarks. States are mapped to hash codes, which allows to count their occurrences with a hash table. These counts are then used to compute a reward bonus according to the classic count-based exploration theory. We find that simple hash functions can achieve surprisingly good results on many challenging tasks. Furthermore, we show that a domain-dependent learned hash code may further improve these results. Detailed analysis reveals important aspects of a good hash function: 1) having appropriate granularity and 2) encoding information relevant to solving the MDP. This exploration strategy achieves near state-of-the-art performance on both continuous control tasks and Atari 2600 games, hence providing a simple yet powerful baseline for solving MDPs that require considerable exploration.

Motivation & Objective

• Motivate robust exploration for deep RL in high-dimensional spaces.
• Generalize count-based exploration to continuous/complex state spaces using hashing.
• Evaluate simple hash-based exploration across challenging deep RL benchmarks.
• Analyze factors that make hashing-based exploration effective (granularity, relevant information).
• Provide a fast, flexible baseline compatible with common DRL algorithms.

Proposed method

• Discretize state space with a hash function to obtain counts n(φ(s)) and add a bonus β / sqrt(n(φ(s))) to the reward.
• Use SimHash as a practical, scalable hashing method for continuous states (Algorithm 1).
• Explore learned hashing by training an autoencoder to produce binary codes, then apply SimHash to obtain φ(s) (Algorithm 2).
• Optionally preprocess states with domain knowledge (static features like BASS) or learned representations to improve hashing quality.
• Evaluate with TRPO on rllab continuous control benchmarks and Atari 2600 games, comparing against baselines and prior exploration methods.

Experimental results

Research questions

• RQ1 Can count-based exploration via hashing improve performance across continuous control and Atari benchmarks?
• RQ2 How do static vs learned hash codes affect exploration performance in image-based observations?
• RQ3 What hash-function properties (granularity, informative encoding) contribute most to effective exploration?
• RQ4 How does hashing-based exploration compare with state-of-the-art deep RL exploration methods?

Key findings

• Hash-based exploration with hashing achieves near state-of-the-art performance on several benchmarks.
• Static SimHash, and learned hashing (AE-SimHash) can outperform baseline TRPO on multiple Atari games and continuous control tasks.
• Domain-dependent preprocessing (BASS) or learned hash codes can yield substantial improvements, especially on games like Montezuma’s Revenge and Venture.
• Simple hashing approaches can provide strong exploration signals without complex intrinsic motivation schemes.
• The approach is fast, flexible, and complementary to existing DRL algorithms.