[Paper Review] VIME: Variational Information Maximizing Exploration

Rein Houthooft, Xi Chen+4 more|arXiv (Cornell University)|May 31, 2016

Reinforcement Learning in Robotics39 references376 citations

TL;DR

Introduces VIME, a curiosity-driven exploration strategy for continuous control that uses variational inference in Bayesian neural networks to maximize information gain about environment dynamics, improving exploration over heuristic methods.

ABSTRACT

Scalable and effective exploration remains a key challenge in reinforcement learning (RL). While there are methods with optimality guarantees in the setting of discrete state and action spaces, these methods cannot be applied in high-dimensional deep RL scenarios. As such, most contemporary RL relies on simple heuristics such as epsilon-greedy exploration or adding Gaussian noise to the controls. This paper introduces Variational Information Maximizing Exploration (VIME), an exploration strategy based on maximization of information gain about the agent's belief of environment dynamics. We propose a practical implementation, using variational inference in Bayesian neural networks which efficiently handles continuous state and action spaces. VIME modifies the MDP reward function, and can be applied with several different underlying RL algorithms. We demonstrate that VIME achieves significantly better performance compared to heuristic exploration methods across a variety of continuous control tasks and algorithms, including tasks with very sparse rewards.

Motivation & Objective

• Address exploration in high-dimensional continuous reinforcement learning environments.
• Maximize information gain about environment dynamics to guide exploration.
• Use variational inference with Bayesian neural networks to compute intrinsic rewards.
• Demonstrate effectiveness across multiple RL algorithms and tasks, including sparse rewards.

Proposed method

• Formalize curiosity as the mutual information between next state and the dynamics model parameters given history.
• Approximate posterior updates with variational Bayes using a Bayesian neural network as the dynamics model.
• Define intrinsic reward as an information gain term: η DKL[q(θ; φt+1) || q(θ; φt)].
• Implement a practical SGVB (Bayes by Backprop) training routine with a fully factorized Gaussian posterior for θ.
• Update the posterior periodically using a replay pool to stabilize learning and enable efficient computation of the intrinsic reward.
• Integrate VIME with standard RL algorithms (e.g., TRPO, REINFORCE, ERWR) to improve exploration in continuous control tasks.

Experimental results

Research questions

• RQ1 Does VIME improve exploration and final performance on continuous control tasks with sparse rewards?
• RQ2 Is VIME effective across different underlying RL algorithms beyond TRPO?
• RQ3 How does the exploration parameter η affect the balance between exploration and exploitation?
• RQ4 Can variational Bayesian dynamics models scale to high-dimensional continuous control without discretizing state-action spaces?

Key findings

• VIME significantly outperforms naive exploration strategies on several sparse-reward continuous control tasks (e.g., MountainCar, CartPoleSwingup, HalfCheetah).
• VIME yields performance gains when paired with TRPO, REINFORCE, and ERWR across multiple domains.
• The method enables learning on challenging tasks with sparse rewards, including the hierarchical SwimmerGather task.
• Visitation patterns under VIME show more diffused exploration compared to Gaussian noise, indicating systematic exploration.
• There exists a wide range of η values for which MountainCar is effectively solved across different algorithms, indicating robustness of the exploration signal.