[Paper Review] UCB Exploration via Q-Ensembles

Richard Y. Chen, Szymon Sidor+2 more|arXiv (Cornell University)|Jun 5, 2017

Reinforcement Learning in RoboticsComputer Science24 references77 citations

TL;DR

This paper introduces Q-ensembles with an upper-confidence bound (UCB) exploration strategy for deep Q-learning, showing improved Atari game performance over Double DQN and Bootstrapped DQN.

ABSTRACT

We show how an ensemble of $Q^*$-functions can be leveraged for more effective exploration in deep reinforcement learning. We build on well established algorithms from the bandit setting, and adapt them to the $Q$-learning setting. We propose an exploration strategy based on upper-confidence bounds (UCB). Our experiments show significant gains on the Atari benchmark.

Motivation & Objective

• Motivate improved exploration in deep Q-learning beyond standard ε-greedy and Boltzmann methods.
• Develop a tractable Bayesian-inspired framework by maintaining a Q-function ensemble to approximate the posterior over Q*.
• Propose an Ensemble Voting mechanism for action selection using multiple Q-heads.
• Introduce a UCB-based exploration strategy that uses ensemble uncertainty to drive exploration.
• Empirically validate the approach on Atari benchmarks against competitive baselines.

Proposed method

• Construct K independently initialized Q* function heads to form a Q-ensemble (Q1,...,QK).
• Use an experience replay buffer and standard Bellman updates with a target network for stability.
• Aggregate actions by Majority Vote over the argmax actions from each Qk (Algorithm 1).
• Derive a UCB-based exploration rule by adding the empirical standard deviation of the ensemble to the mean Q-values (Equation 13).
• Expose two exploration strategies: Ensemble Voting (Algorithm 1) and UCB Exploration with Q-Ensembles (Algorithm 2).
• Empirically evaluate on 40M frames across Atari games, comparing to Double DQN, Bootstrapped DQN, and a count-based baseline (A3C+).

Experimental results

Research questions

• RQ1 Does Ensemble Voting improve over Double DQN and Bootstrapped DQN on Atari games?
• RQ2 Does the UCB exploration strategy with Q-ensembles further improve learning beyond Ensemble Voting?
• RQ3 How do Q-ensembles with UCB exploration compare to count-based exploration methods like A3C+ across Atari game categories?
• RQ4 In which game categories (Human Optimal, Score Explicit, Dense Reward, Sparse Reward) do these methods excel?
• RQ5 What is the impact of ensemble size and exploration hyperparameters on performance?

Key findings

• Ensemble Voting outperforms both Double DQN and Bootstrapped DQN on Atari games without explicit exploration.
• UCB exploration with Q-ensembles yields further improvements over Ensemble Voting, achieving top performance in many games.
• Across 49 Atari games, UCB exploration attains the highest maximal mean reward in 30 games compared to the baselines.
• When compared to A3C+ trained with 200M frames, the proposed methods (40M frames) achieve higher average rewards in a substantial number of games.
• Overall, UCB exploration improves performance across multiple game categories, including Human Optimal, Score Explicit, and Dense Reward.