[Paper Review] Learning more skills through optimistic exploration

DJ Strouse, Kate Baumli+3 more|arXiv (Cornell University)|Jul 29, 2021

Reinforcement Learning in Robotics35 references4 citations

TL;DR

This paper proposes DISDAIN, an intrinsic exploration bonus that uses ensemble disagreement among discriminators to estimate epistemic uncertainty and encourage exploration in unsupervised skill discovery. By rewarding policy behavior that reduces uncertainty in the discriminator's predictions—especially in novel states—DISDAIN overcomes pessimistic exploration, significantly improving skill learning in both grid worlds and Atari games compared to baseline methods.

ABSTRACT

Unsupervised skill learning objectives (Gregor et al., 2016, Eysenbach et al., 2018) allow agents to learn rich repertoires of behavior in the absence of extrinsic rewards. They work by simultaneously training a policy to produce distinguishable latent-conditioned trajectories, and a discriminator to evaluate distinguishability by trying to infer latents from trajectories. The hope is for the agent to explore and master the environment by encouraging each skill (latent) to reliably reach different states. However, an inherent exploration problem lingers: when a novel state is actually encountered, the discriminator will necessarily not have seen enough training data to produce accurate and confident skill classifications, leading to low intrinsic reward for the agent and effective penalization of the sort of exploration needed to actually maximize the objective. To combat this inherent pessimism towards exploration, we derive an information gain auxiliary objective that involves training an ensemble of discriminators and rewarding the policy for their disagreement. Our objective directly estimates the epistemic uncertainty that comes from the discriminator not having seen enough training examples, thus providing an intrinsic reward more tailored to the true objective compared to pseudocount-based methods (Burda et al., 2019). We call this exploration bonus discriminator disagreement intrinsic reward, or DISDAIN. We demonstrate empirically that DISDAIN improves skill learning both in a tabular grid world (Four Rooms) and the 57 games of the Atari Suite (from pixels). Thus, we encourage researchers to treat pessimism with DISDAIN.

Motivation & Objective

• To address the inherent 'pessimistic exploration' problem in variational skill discovery, where novel states lead to low-confidence discriminator predictions and thus discourage exploration.
• To distinguish between aleatoric uncertainty (poor policy performance) and epistemic uncertainty (lack of training data), with the latter being a desirable signal for exploration.
• To design an intrinsic exploration bonus tailored to skill discovery that directly estimates and rewards information gain about the discriminator's parameters.
• To empirically validate that the proposed DISDAIN bonus improves the number and diversity of skills learned in both tabular and pixel-based environments.

Proposed method

• Train an ensemble of parametric skill discriminators to estimate epistemic uncertainty via prediction disagreement across members.
• Define the DISDAIN bonus as the negative entropy of the ensemble-averaged discriminator predictions, which increases when predictions are uncertain and diverse.
• Use the DISDAIN bonus as an intrinsic reward signal to optimize the policy, encouraging it to explore states where the ensemble disagrees.
• Formally derive DISDAIN from a Bayesian perspective, modeling the posterior over discriminator parameters using an ensemble and maximizing information gain.
• Integrate DISDAIN into existing variational infomax skill learning frameworks, such as DIAYN, without modifying the core skill learning objective.
• Tune hyperparameters like the ensemble size and bonus weighting via ablation to balance intrinsic and extrinsic rewards.

Experimental results

Research questions

• RQ1 Can ensemble-based disagreement in discriminators serve as a reliable signal for epistemic uncertainty during skill discovery?
• RQ2 Does using DISDAIN as an intrinsic bonus improve the number and diversity of skills learned in unsupervised skill discovery?
• RQ3 How does DISDAIN compare to pseudocount-based and RND-based exploration bonuses in skill learning performance?
• RQ4 Does DISDAIN maintain performance across different hyperparameter settings and environments?

Key findings

• DISDAIN significantly improves skill learning in the Four Rooms grid world, with a 25% increase in the number of distinct skills discovered compared to unbonused training.
• On the Atari 2600 suite, DISDAIN achieves higher zero-shot reward performance and improved lifetime state coverage compared to unbonused and RND-augmented baselines.
• RND fails to improve skill learning and even harms performance when its bonus weight is increased, indicating incompatibility with non-stationary skill learning objectives.
• The 'ensemble-only' baseline without the DISDAIN bonus shows minimal improvement, confirming that the intrinsic bonus mechanism is essential, not just the ensembling itself.
• DISDAIN maintains strong performance across a wide range of hyperparameter settings, including ensemble size and bonus weighting, demonstrating robustness.
• Qualitative analysis shows that DISDAIN leads to a higher probability of improvement in zero-shot evaluation and better distribution of percentile improvements across Atari games.