[Paper Review] Implicit Generative Modeling for Efficient Exploration

Neale Ratzlaff, Qinxun Bai+2 more|arXiv (Cornell University)|Nov 19, 2019

Reinforcement Learning in Robotics43 references4 citations

TL;DR

This paper proposes a novel implicit generative modeling approach using amortized Stein Variational Gradient Descent to approximate the posterior distribution over environment dynamics in deep reinforcement learning. By sampling multiple neural network dynamics models from the generator, the method estimates Bayesian uncertainty through prediction variance, which serves as an intrinsic reward for efficient exploration. It outperforms state-of-the-art methods in data efficiency on challenging sparse-reward environments.

ABSTRACT

Efficient exploration remains a challenging problem in reinforcement learning, especially for those tasks where rewards from environments are sparse. A commonly used approach for exploring such environments is to introduce some "intrinsic" reward. In this work, we focus on model uncertainty estimation as an intrinsic reward for efficient exploration. In particular, we introduce an implicit generative modeling approach to estimate a Bayesian uncertainty of the agent's belief of the environment dynamics. Each random draw from our generative model is a neural network that instantiates the dynamic function, hence multiple draws would approximate the posterior, and the variance in the future prediction based on this posterior is used as an intrinsic reward for exploration. We design a training algorithm for our generative model based on the amortized Stein Variational Gradient Descent. In experiments, we compare our implementation with state-of-the-art intrinsic reward-based exploration approaches, including two recent approaches based on an ensemble of dynamic models. In challenging exploration tasks, our implicit generative model consistently outperforms competing approaches regarding data efficiency in exploration.

Motivation & Objective

• To address the challenge of sample inefficiency in deep reinforcement learning, especially in environments with sparse rewards.
• To improve exploration efficiency by modeling uncertainty in the agent's belief of environment dynamics.
• To develop a flexible, non-parametric posterior approximation for dynamics models without restrictive distributional assumptions.
• To outperform existing intrinsic reward-based exploration methods in data efficiency and exploration coverage.
• To evaluate the method's transferability to downstream dense-reward tasks through policy transfer.

Proposed method

• A generator network is trained to implicitly represent the posterior distribution over dynamics model parameters using amortized Stein Variational Gradient Descent (SVGD).
• Each random draw from the generator produces a neural network that acts as a dynamics model, enabling multiple samples to approximate the posterior distribution.
• Prediction variance across these sampled dynamics models is used as an intrinsic reward signal to encourage exploration in uncertain state-action regions.
• The method avoids parametric assumptions and ELBO approximations by directly minimizing KL divergence between the implicit posterior and true posterior.
• The framework allows unlimited sampling after a single training phase, unlike ensemble methods that require retraining for each sample.
• The approach is evaluated in both pure exploration and downstream task settings, including policy transfer experiments with SAC.

Experimental results

Research questions

• RQ1 Can implicit generative modeling with amortized SVGD provide a more flexible and efficient posterior approximation for dynamics models than ensemble-based methods?
• RQ2 Does uncertainty estimated from an implicit posterior distribution lead to better data efficiency in exploration compared to existing intrinsic reward methods?
• RQ3 Can a pre-trained exploration policy based on this uncertainty estimation significantly improve downstream task learning in dense-reward environments?
• RQ4 How does the length of the pure exploration phase affect downstream task performance in policy transfer settings?
• RQ5 Is the proposed method robust and effective in environments with sparse external rewards?

Key findings

• The proposed method consistently outperforms state-of-the-art intrinsic reward-based exploration methods in terms of data efficiency across three challenging exploration tasks.
• In policy transfer experiments, initializing SAC with the exploration policy from this method led to superior performance at 1M training steps, even outperforming methods like MAX, ICM, and Disagreement.
• Even with only 4,000 steps of pure exploration, the method improved downstream task performance over the SAC baseline, demonstrating strong sample efficiency.
• The method showed a clear performance margin over all baselines in both warm-up and non-warm-up settings, confirming its robustness and transferability.
• The ablation study confirmed that longer pure exploration phases further improved downstream task learning, with performance increasing monotonically with exploration duration.
• The results demonstrate that implicit posterior modeling via amortized SVGD enables more effective uncertainty estimation for exploration than ensemble-based disagreement or uncertainty measures.