[Paper Review] Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networks

Litian Liang, Yaosheng Xu+5 more|arXiv (Cornell University)|Sep 16, 2022

Mental Health Research Topics4 citations

TL;DR

MeanQ proposes a simple ensemble deep reinforcement learning method that reduces temporal-difference value estimation variance by averaging predictions from five independent deep Q-networks. By using ensemble averaging, MeanQ achieves state-of-the-art sample efficiency, outperforming baselines like SUNRISE and Rainbow DQN, and attains average human-level performance using only 200K interaction steps on the Atari benchmark.

ABSTRACT

In temporal-difference reinforcement learning algorithms, variance in value estimation can cause instability and overestimation of the maximal target value. Many algorithms have been proposed to reduce overestimation, including several recent ensemble methods, however none have shown success in sample-efficient learning through addressing estimation variance as the root cause of overestimation. In this paper, we propose MeanQ, a simple ensemble method that estimates target values as ensemble means. Despite its simplicity, MeanQ shows remarkable sample efficiency in experiments on the Atari Learning Environment benchmark. Importantly, we find that an ensemble of size 5 sufficiently reduces estimation variance to obviate the lagging target network, eliminating it as a source of bias and further gaining sample efficiency. We justify intuitively and empirically the design choices in MeanQ, including the necessity of independent experience sampling. On a set of 26 benchmark Atari environments, MeanQ outperforms all tested baselines, including the best available baseline, SUNRISE, at 100K interaction steps in 16/26 environments, and by 68% on average. MeanQ also outperforms Rainbow DQN at 500K steps in 21/26 environments, and by 49% on average, and achieves average human-level performance using 200K ($\pm$100K) interaction steps. Our implementation is available at https://github.com/indylab/MeanQ.

Motivation & Objective

• Address the instability and overestimation in deep Q-learning caused by high variance in value function estimation.
• Investigate whether ensemble averaging can reduce variance well enough to eliminate the need for a target network, a known source of bias.
• Demonstrate that variance reduction through ensemble averaging leads to superior sample efficiency and performance compared to existing methods.
• Clarify the role of key design choices—shared exploration, shared replay buffer, and independent replay sampling—in enabling effective ensemble learning.
• Rule out alternative explanations for performance gains, such as model size or increased update frequency, by ablation studies.

Proposed method

• Use an ensemble of five independent deep Q-networks to estimate state–action values, with target values computed as the mean of the ensemble predictions.
• Apply a shared exploration policy derived from the ensemble mean to ensure consistent behavior during experience collection.
• Maintain a single shared replay buffer for all ensemble members to improve data efficiency and experience diversity.
• Implement independent experience sampling from the replay buffer to preserve ensemble diversity and prevent co-adaptation.
• Train all ensemble members simultaneously using stochastic gradient descent with experience replay, minimizing the TD error on bootstrapped targets.
• Eliminate the lagging target network by relying on ensemble averaging to stabilize learning and reduce variance.

Experimental results

Research questions

• RQ1 Can ensemble averaging of deep Q-networks sufficiently reduce value estimation variance to eliminate the need for a target network?
• RQ2 How does independent replay sampling affect the performance and stability of ensemble-based value estimation?
• RQ3 To what extent do performance gains in MeanQ stem from variance reduction rather than increased model capacity or more frequent updates?
• RQ4 How does MeanQ compare to state-of-the-art baselines like SUNRISE and Rainbow DQN in terms of sample efficiency and final performance?
• RQ5 What is the impact of shared exploration and shared replay on the effectiveness of the ensemble method?

Key findings

• MeanQ outperforms SUNRISE at 100K interaction steps in 16 out of 26 Atari environments and achieves a 68% average improvement in normalized return.
• At 500K steps, MeanQ surpasses Rainbow DQN in 21 out of 26 environments and achieves a 49% higher average normalized return.
• MeanQ reaches average human-level performance using only 200K ± 100K interaction steps across the 26 Atari environments.
• The ensemble of size 5 reduces estimation variance sufficiently to eliminate the need for a target network, which further improves sample efficiency.
• Ablation studies show that performance gains in MeanQ are not due to model size or increased update frequency, but specifically due to variance reduction via ensemble averaging.
• Independent replay sampling is critical: without it, ensemble diversity degrades and performance drops significantly, highlighting its necessity for effective variance reduction.