[Paper Review] EX2: Exploration with Exemplar Models for Deep Reinforcement Learning

Justin Fu, John D. Co-Reyes+1 more|arXiv (Cornell University)|Mar 3, 2017

Reinforcement Learning in RoboticsComputer Science61 citations

TL;DR

EX2 introduces exemplar discriminators to estimate implicit state density for intrinsic rewards, enabling effective exploration without explicit generative models, including strong results on vizDoom and competitive performance on Atari and continuous control tasks.

ABSTRACT

Deep reinforcement learning algorithms have been shown to learn complex tasks using highly general policy classes. However, sparse reward problems remain a significant challenge. Exploration methods based on novelty detection have been particularly successful in such settings but typically require generative or predictive models of the observations, which can be difficult to train when the observations are very high-dimensional and complex, as in the case of raw images. We propose a novelty detection algorithm for exploration that is based entirely on discriminatively trained exemplar models, where classifiers are trained to discriminate each visited state against all others. Intuitively, novel states are easier to distinguish against other states seen during training. We show that this kind of discriminative modeling corresponds to implicit density estimation, and that it can be combined with count-based exploration to produce competitive results on a range of popular benchmark tasks, including state-of-the-art results on challenging egocentric observations in the vizDoom benchmark.

Motivation & Objective

• Address sparse reward exploration in deep RL without relying on generative observation models.
• Develop discriminatively trained exemplar models that estimate state novelty as implicit density.
• Amortize exemplar training to scale to large state spaces and high-dimensional observations.
• Integrate the novelty signal with count-based exploration bonuses.
• Demonstrate effectiveness across low-dimensional benchmarks and complex image-based domains like vizDoom and Atari.

Proposed method

• Train an exemplar model comprising discriminators D_x* that distinguish each visited state x* from all other states.
• Show that the exemplar discriminators induce an implicit density estimate for P_X(x) via D_x*(x*) and smoothing via noise in latent space.
• Amortize training with an encoder-based, conditioned discriminator (amortized/exemplar-conditioned network).
• Introduce K-Exemplar and amortized architectures to share features and reduce per-state classifier count.
• Incorporate a novelty bonus into the RL reward of the form R'(s,a)=R(s,a)+β f(D_s(s)), with f(D_s(s)) linked to the estimated density.
• Relate the method to pseudo-count exploration and discuss connections to KDE and GANs.

Experimental results

Research questions

• RQ1 Can discriminatively trained exemplar models estimate state novelty without explicit generative models?
• RQ2 Do exemplar-based implicit density estimates provide effective intrinsic rewards for exploration in high-dimensional visual tasks?
• RQ3 Can amortized or K-exemplar architectures scale exemplar-based exploration to large state spaces?
• RQ4 How does EX2 perform relative to existing generative-model-based exploration methods on Atari and vizDoom?
• RQ5 What is the impact of latent-space smoothing on density estimation and exploration bonuses?

Key findings

• EX2 matches or exceeds prior explicit density estimation methods on several tasks and greatly improves performance on vizDoom DoomMyWayHome+.
• On SwimmerGather and SparseHalfCheetah, EX2 and derivatives (VIME, hashing) significantly outperform naïve TRPO and KDE.
• In Atari games, EX2 variants achieve competitive performance with prior density-based methods; on DoomMyWayHome+ EX2 greatly outperforms all prior exploration techniques.
• Implicit density estimation via exemplar models provides robust exploration bonuses in high-dimensional image domains where explicit generative models struggle.
• Amortized and K-Exemplar architectures offer scalable, effective density estimation with shared representations across exemplars.