[Paper Review] Dropout Q-Functions for Doubly Efficient Reinforcement Learning

T. HIRAOKA, Takahisa Imagawa+3 more|arXiv (Cornell University)|Oct 5, 2021

Reinforcement Learning in Robotics39 references19 citations

TL;DR

This paper proposes DroQ, a computationally efficient variant of REDQ that uses a small ensemble of dropout Q-functions with layer normalization. Despite its simplicity, DroQ achieves sample efficiency comparable to REDQ and computational efficiency over twice that of REDQ, outperforming SAC in sample efficiency while matching its speed.

ABSTRACT

Randomized ensembled double Q-learning (REDQ) (Chen et al., 2021b) has recently achieved state-of-the-art sample efficiency on continuous-action reinforcement learning benchmarks. This superior sample efficiency is made possible by using a large Q-function ensemble. However, REDQ is much less computationally efficient than non-ensemble counterparts such as Soft Actor-Critic (SAC) (Haarnoja et al., 2018a). To make REDQ more computationally efficient, we propose a method of improving computational efficiency called DroQ, which is a variant of REDQ that uses a small ensemble of dropout Q-functions. Our dropout Q-functions are simple Q-functions equipped with dropout connection and layer normalization. Despite its simplicity of implementation, our experimental results indicate that DroQ is doubly (sample and computationally) efficient. It achieved comparable sample efficiency with REDQ, much better computational efficiency than REDQ, and comparable computational efficiency with that of SAC.

Motivation & Objective

• Address the computational inefficiency of REDQ, which uses large Q-function ensembles despite high sample efficiency.
• Improve computational efficiency without sacrificing sample efficiency in continuous-control reinforcement learning.
• Explore whether dropout can serve as a viable alternative to ensemble methods in high UTD ratio settings.
• Investigate engineering practices—specifically dropout and layer normalization—that enhance bias reduction and sample efficiency in high UTD RL.
• Demonstrate that a minimal architectural change (dropout + normalization) can yield doubly efficient RL methods.

Proposed method

• Propose DroQ, a variant of REDQ that uses a small ensemble (M=2) of Q-functions with dropout and layer normalization.
• Apply dropout during forward passes in Q-networks to create stochastic ensembles, replacing the full ensemble of independent networks in REDQ.
• Integrate layer normalization into each fully connected layer of the Q-networks to stabilize training and improve generalization.
• Use in-target minimization with M=2 for both policy and Q-function updates, similar to REDQ and SAC.
• Train the policy using the minimum of the two dropout Q-functions, which acts as a regularized estimate of the Q-value.
• Implement the method by adding only dropout and layer normalization layers to existing SAC or REDQ codebases, ensuring simplicity and reproducibility.

Experimental results

Research questions

• RQ1 Can a small ensemble of dropout-based Q-functions achieve sample efficiency comparable to REDQ’s large ensemble?
• RQ2 Does the combination of dropout and layer normalization improve bias reduction and sample efficiency in high UTD ratio RL?
• RQ3 Can dropout-based Q-functions match the computational efficiency of SAC while surpassing REDQ in speed?
• RQ4 What engineering practices (e.g., normalization, ensemble size) are critical for effective dropout use in high UTD RL?
• RQ5 Is dropout a viable alternative to full ensembles in high UTD reinforcement learning settings?

Key findings

• DroQ achieves sample efficiency comparable to REDQ across MuJoCo benchmarks, with average returns matching or exceeding those of REDQ.
• DroQ reduces computational time per update by more than 2x compared to REDQ, making it significantly faster than the original REDQ.
• DroQ matches the computational efficiency of SAC, achieving similar wallclock time per update while outperforming SAC in sample efficiency.
• The ablation study confirms that combining dropout with ensemble averaging (M=2) and layer normalization is essential for bias reduction and performance.
• Layer normalization significantly improves training stability and performance, outperforming batch normalization and group normalization in the same setup.
• The method is highly practical: DroQ can be implemented with just a few lines of code by adding dropout and layer normalization to existing SAC or REDQ implementations.