[Paper Review] Explicit Explore-Exploit Algorithms in Continuous State Spaces

Mikael Henaff|arXiv (Cornell University)|Nov 1, 2019

Reinforcement Learning in Robotics14 citations

TL;DR

This paper proposes a model-based reinforcement learning algorithm that explicitly separates exploration and exploitation phases in continuous or large state spaces by maintaining an ensemble of dynamics models. It explores by maximizing disagreement between model predictions and exploits using refined models, achieving provably sample-efficient learning with polynomial sample complexity in a structural complexity measure, validated empirically using neural networks.

ABSTRACT

We present a new model-based algorithm for reinforcement learning (RL) which consists of explicit exploration and exploitation phases, and is applicable in large or infinite state spaces. The algorithm maintains a set of dynamics models consistent with current experience and explores by finding policies which induce high disagreement between their state predictions. It then exploits using the refined set of models or experience gathered during exploration. We show that under realizability and optimal planning assumptions, our algorithm provably finds a near-optimal policy with a number of samples that is polynomial in a structural complexity measure which we show to be low in several natural settings. We then give a practical approximation using neural networks and demonstrate its performance and sample efficiency in practice.

Motivation & Objective

• To develop a sample-efficient reinforcement learning algorithm for large or continuous state spaces where tabular methods fail.
• To enable systematic exploration by identifying policies that maximize disagreement among a set of consistent dynamics models.
• To prove that the algorithm finds a near-optimal policy with sample complexity polynomial in a structural complexity measure (model misfit matrix rank).
• To provide a practical implementation using neural networks that maintains sample efficiency in real-world environments.

Proposed method

• Maintains a set of dynamics models consistent with observed experience, updating them as new data is collected.
• Performs exploration by solving a fictitious MDP where the state is the concatenation of model states and rewards are the distributional disagreement (e.g., KL divergence) between model predictions.
• Uses a distance metric δ (e.g., KL divergence or total variation) to quantify disagreement between model predictions over time steps.
• Switches to exploitation when no policy induces sufficient disagreement, selecting a single model to plan for maximum cumulative reward.
• Employs neural networks to approximate the dynamics models and uses DQN or PPO for policy optimization in the exploitation phase.
• Uses a replay buffer to store trajectories from exploration and fine-tunes models using supervised learning on collected data.

Experimental results

Research questions

• RQ1 Can explicit exploration based on model disagreement achieve sample-efficient learning in continuous or large state spaces?
• RQ2 What structural complexity measure governs the sample complexity of such an algorithm under realizability and optimal planning assumptions?
• RQ3 How does model disagreement between ensemble members guide effective exploration in high-dimensional or infinite state spaces?
• RQ4 Can a neural network-based approximation of the algorithm maintain sample efficiency in practice?
• RQ5 How does the algorithm compare to intrinsic curiosity or RND-based exploration in continuous control and navigation tasks?

Key findings

• The algorithm achieves sample complexity polynomial in the rank of the model misfit matrix, which is low in natural settings such as factored MDPs and low-rank MDPs.
• In the maze environment, the Neural-E3 agent with DQN exploitation outperformed MCTS-based planning, especially under misleading reward shaping, due to reduced bias in search.
• On continuous control tasks, Neural-E3 with DQN achieved strong performance, with the exploit phase trained on 750,000 DQN updates using experience from exploration.
• The model disagreement metric effectively identified unexplored regions, with models agreeing up to step 28 but diverging at step 29 in the maze task, signaling a critical exploration point.
• The algorithm demonstrated robustness to reward shaping and outperformed baselines in environments with sparse or deceptive rewards.
• Hyperparameter tuning showed that prioritized replay and proper learning rates were critical for stable training, while parameter noise did not improve performance.