[Paper Review] Weighted QMIX: Expanding Monotonic Value Function Factorisation.

Tabish Rashid, Gregory Farquhar+2 more|arXiv (Cornell University)|Jun 18, 2020

Reinforcement Learning in RoboticsComputer Science12 references22 citations

TL;DR

This paper proposes Weighted QMIX, a novel extension of QMIX that improves representational capacity by introducing weighted projections during value function factorization. By prioritizing high-quality joint actions through adaptive weighting schemes—Centrally-Weighted (CW) and Optimistically-Weighted (OW) QMIX—it enables exact recovery of optimal policies even when QMIX's standard unweighted projection fails, demonstrating superior performance on predator-prey and StarCraft benchmarks.

ABSTRACT

QMIX is a popular $Q$-learning algorithm for cooperative MARL in the centralised training and decentralised execution paradigm. In order to enable easy decentralisation, QMIX restricts the joint action $Q$-values it can represent to be a monotonic mixing of each agent's utilities. However, this restriction prevents it from representing value functions in which an agent's ordering over its actions can depend on other agents' actions. To analyse this representational limitation, we first formalise the objective QMIX optimises, which allows us to view QMIX as an operator that first computes the $Q$-learning targets and then projects them into the space representable by QMIX. This projection returns a representable $Q$-value that minimises the unweighted squared error across all joint actions. We show in particular that this projection can fail to recover the optimal policy even with access to $Q^*$, which primarily stems from the equal weighting placed on each joint action. We rectify this by introducing a weighting into the projection, in order to place more importance on the better joint actions. We propose two weighting schemes and prove that they recover the correct maximal action for any joint action $Q$-values, and therefore for $Q^*$ as well. Based on our analysis and results in the tabular setting we introduce two scalable versions of our algorithm, Centrally-Weighted (CW) QMIX and Optimistically-Weighted (OW) QMIX and demonstrate improved performance on both predator-prey and challenging multi-agent StarCraft benchmark tasks.

Motivation & Objective

• To identify and address the representational limitation of QMIX, which arises from its unweighted projection that fails to recover optimal policies even with access to Q*.
• To formalize QMIX's optimization objective as a projection operator that minimizes unweighted squared error across joint actions.
• To improve the projection step by introducing weights that prioritize better joint actions, thereby enhancing policy recovery.
• To develop scalable, effective variants—CW QMIX and OW QMIX—that maintain the centralised training and decentralized execution paradigm while improving performance.

Proposed method

• The paper formalizes QMIX as an operator that computes Q-learning targets and projects them into the QMIX representable space via unweighted least-squares minimization.
• It identifies that the equal weighting of all joint actions in the projection leads to suboptimal policy recovery, even when Q* is known.
• It introduces a weighted projection mechanism that assigns higher importance to better-performing joint actions during the projection step.
• Two weighting schemes are proposed: Centrally-Weighted QMIX, which uses centralized knowledge of joint action quality, and Optimistically-Weighted QMIX, which uses optimistic estimates to prioritize high-return actions.
• Theoretical analysis proves that both weighting schemes recover the correct maximal action for any joint action Q-values, including Q*.
• The methods are extended to scalable deep reinforcement learning settings and evaluated on tabular and deep MARL benchmarks.

Experimental results

Research questions

• RQ1 Can the unweighted projection in QMIX lead to suboptimal policy recovery even when Q* is available?
• RQ2 How does the equal weighting of joint actions in QMIX's projection affect its ability to represent optimal policies?
• RQ3 Can introducing adaptive weighting in the projection step improve policy recovery and performance in cooperative MARL?
• RQ4 Do the proposed weighting schemes—CW QMIX and OW QMIX—achieve exact recovery of the optimal policy for any Q-value function?
• RQ5 Does the improved representational capacity translate to better performance in challenging multi-agent environments like StarCraft and predator-prey?

Key findings

• The standard QMIX projection fails to recover the optimal policy even when given access to Q*, due to the equal weighting of all joint actions in the unweighted least-squares projection.
• Weighted QMIX, through its adaptive weighting mechanism, successfully recovers the correct maximal action for any joint action Q-values, including Q*.
• CW QMIX and OW QMIX both achieve exact policy recovery under the theoretical conditions of the analysis.
• In tabular environments, the proposed methods outperform standard QMIX by correctly identifying optimal actions where QMIX fails.
• On deep MARL benchmarks, including predator-prey and StarCraft II, Weighted QMIX shows improved sample efficiency and final performance compared to standard QMIX.
• The improvements are attributed to the enhanced representational capacity from prioritizing high-quality joint actions during value function projection.