[Paper Review] Maxmin Q-learning: Controlling the Estimation Bias of Q-learning

Qingfeng Lan, Yangchen Pan+2 more|arXiv (Cornell University)|Feb 16, 2020

Reinforcement Learning in Robotics13 references38 citations

TL;DR

Maxmin Q-learning generalizes Q-learning by using the minimum over N action-value estimators in the bootstrap target to flexibly control estimation bias and variance, with convergence guarantees and empirical validation.

ABSTRACT

Q-learning suffers from overestimation bias, because it approximates the maximum action value using the maximum estimated action value. Algorithms have been proposed to reduce overestimation bias, but we lack an understanding of how bias interacts with performance, and the extent to which existing algorithms mitigate bias. In this paper, we 1) highlight that the effect of overestimation bias on learning efficiency is environment-dependent; 2) propose a generalization of Q-learning, called \emph{Maxmin Q-learning}, which provides a parameter to flexibly control bias; 3) show theoretically that there exists a parameter choice for Maxmin Q-learning that leads to unbiased estimation with a lower approximation variance than Q-learning; and 4) prove the convergence of our algorithm in the tabular case, as well as convergence of several previous Q-learning variants, using a novel Generalized Q-learning framework. We empirically verify that our algorithm better controls estimation bias in toy environments, and that it achieves superior performance on several benchmark problems.

Motivation & Objective

• Motivate the study of estimation bias in Q-learning and its environment-dependent impact on learning efficiency.
• Introduce Maxmin Q-learning as a generalization that can adjust bias and variance via the number of estimators N.
• Provide theoretical analysis showing unbiased estimation and reduced variance for appropriate N.
• Prove convergence of Maxmin Q-learning in the tabular setting within a Generalized Q-learning framework.
• Empirically validate bias control and performance across toy environments and benchmark problems.

Proposed method

• Maintain N action-value estimates Q^1, ..., Q^N for each state-action pair.
• Use the minimum across the N estimates in the bootstrap target: max_a' min_i Q^i(s', a').
• Update a randomly chosen subset of estimators per step using a replay buffer and mini-batches.
• Optionally incorporate target networks for Maxmin DQN by maintaining a target for each estimator.
• Provide theoretical results: E[Z_MN] (bias) decreases with N and Var[Q_sa^min] decreases with N.
• Frame Maxmin Q-learning as a special case of Generalized Q-learning with G(Q) = max_a min_i Q^i_sa.

Experimental results

Research questions

• RQ1 How does overestimation bias in Q-learning affect learning efficiency across different environments?
• RQ2 Can a flexible bias-controlling mechanism reduce both bias and variance of bootstrap targets?
• RQ3 What is the theoretical relationship between the number of estimators N and estimation bias/variance in Maxmin Q-learning?
• RQ4 Does Maxmin Q-learning converge in the tabular setting, and can it generalize to other Q-learning variants under a Generalized Q-learning framework?
• RQ5 Is Maxmin Q-learning practically effective in deep RL benchmarks compared to existing bias-reduction methods?

Key findings

• Maxmin Q-learning reduces overestimation bias as N increases and can induce underestimation bias for large N.
• Var[Q_sa^min] decreases with N and can be lower than the variance of a single estimator using all data for N ≥ 8 under certain conditions.
• There exists an N that yields near-unbiased estimation with lower approximation variance than standard Q-learning.
• Maxmin Q-learning converges in the tabular setting within a Generalized Q-learning framework.
• Empirical results show Maxmin Q-learning robustly controls estimation bias in toy environments and achieves superior performance on several benchmarks.