[Paper Review] Regularized Softmax Deep Multi-Agent $Q$-Learning

Ling Pan, Tabish Rashid+3 more|arXiv (Cornell University)|Mar 22, 2021

Reinforcement Learning in Robotics42 references16 citations

TL;DR

This paper proposes Regularized Softmax (RES) Deep Multi-Agent Q-Learning, a general method to reduce overestimation in multi-agent Q-learning by combining a baseline-based regularization scheme with an efficient approximation of the softmax operator. Applied to QMIX, RES stabilizes learning, prevents unbounded value estimates, and achieves state-of-the-art performance on StarCraft II micromanagement tasks and multi-agent particle environments.

ABSTRACT

Tackling overestimation in $Q$-learning is an important problem that has been extensively studied in single-agent reinforcement learning, but has received comparatively little attention in the multi-agent setting. In this work, we empirically demonstrate that QMIX, a popular $Q$-learning algorithm for cooperative multi-agent reinforcement learning (MARL), suffers from a more severe overestimation in practice than previously acknowledged, and is not mitigated by existing approaches. We rectify this with a novel regularization-based update scheme that penalizes large joint action-values that deviate from a baseline and demonstrate its effectiveness in stabilizing learning. Furthermore, we propose to employ a softmax operator, which we efficiently approximate in a novel way in the multi-agent setting, to further reduce the potential overestimation bias. Our approach, Regularized Softmax (RES) Deep Multi-Agent $Q$-Learning, is general and can be applied to any $Q$-learning based MARL algorithm. We demonstrate that, when applied to QMIX, RES avoids severe overestimation and significantly improves performance, yielding state-of-the-art results in a variety of cooperative multi-agent tasks, including the challenging StarCraft II micromanagement benchmarks.

Motivation & Objective

• To address the severe overestimation problem in deep multi-agent Q-learning, particularly in QMIX, which is underappreciated in prior work.
• To stabilize learning by preventing unbounded growth in joint-action value estimates, which leads to performance degradation.
• To reduce overestimation bias beyond what double estimators or existing regularization can achieve in multi-agent settings.
• To develop an efficient, scalable approximation of the softmax operator for large joint action spaces in MARL.
• To create a general-purpose method applicable to any Q-learning-based MARL algorithm, not limited to QMIX.

Proposed method

• Introduces a regularization-based update that penalizes joint-action Q-values deviating from a baseline, using the discounted return as the baseline to stabilize learning.
• Employs a novel, efficient approximation of the softmax operator over a joint action subspace, with convergence to the true softmax at an exponential rate.
• Derives a modified Bellman loss that combines the regularization penalty and the softmax approximation to reduce overestimation bias.
• Uses a temperature parameter in the softmax approximation to control the trade-off between exploration and exploitation.
• Applies the method to QMIX by replacing its standard Q-learning update with the RES update rule, preserving the monotonic mixing network structure.
• Employs a subspace-based sampling strategy to compute the softmax approximation efficiently without enumerating the full joint action space.

Experimental results

Research questions

• RQ1 Why does QMIX suffer from severe overestimation in practice, despite using Double DQN and Clipped Double Q-learning?
• RQ2 Can existing single-agent overestimation mitigation techniques, such as double estimators or softmax, be effectively adapted to the multi-agent setting?
• RQ3 How can overestimation in the joint-action value function be stabilized when the joint action space grows exponentially with the number of agents?
• RQ4 Can a regularization scheme that penalizes deviation from a baseline improve learning stability and performance in MARL?
• RQ5 Is there an efficient, scalable approximation of the softmax operator that maintains accuracy in large multi-agent action spaces?

Key findings

• RES-QMIX eliminates unbounded value growth in the multi-agent particle environment, where QMIX's value estimates diverge and cause performance collapse.
• RES-QMIX achieves state-of-the-art performance on all StarCraft II micromanagement maps tested, outperforming QMIX, QPLEX, and other SOTA baselines.
• Even without double estimators, RES-QMIX (single) outperforms both standard QMIX and QMIX with double estimators across all maps, demonstrating the effectiveness of regularization alone.
• The value estimates of RES-QMIX are significantly smaller and more stable than those of QMIX, validating the reduction in overestimation bias.
• The proposed softmax approximation converges to the true softmax with exponential speed in the inverse temperature parameter, ensuring high accuracy with low computational cost.
• The method generalizes well: RES can be applied to any Q-learning-based MARL algorithm, not just QMIX, and maintains performance improvements across diverse environments.