[Paper Review] QPLEX: Duplex Dueling Multi-Agent Q-Learning

Jianhao Wang, Zhizhou Ren+3 more|arXiv (Cornell University)|Aug 3, 2020

Reinforcement Learning in Robotics32 references144 citations

TL;DR

QPLEX introduces a duplex dueling network to factorize the joint Q-function under the advantage-based IGM constraint, achieving complete IGM expressiveness and strong performance in online and offline StarCraft II MARL tasks.

ABSTRACT

We explore value-based multi-agent reinforcement learning (MARL) in the popular paradigm of centralized training with decentralized execution (CTDE). CTDE has an important concept, Individual-Global-Max (IGM) principle, which requires the consistency between joint and local action selections to support efficient local decision-making. However, in order to achieve scalability, existing MARL methods either limit representation expressiveness of their value function classes or relax the IGM consistency, which may suffer from instability risk or may not perform well in complex domains. This paper presents a novel MARL approach, called duPLEX dueling multi-agent Q-learning (QPLEX), which takes a duplex dueling network architecture to factorize the joint value function. This duplex dueling structure encodes the IGM principle into the neural network architecture and thus enables efficient value function learning. Theoretical analysis shows that QPLEX achieves a complete IGM function class. Empirical experiments on StarCraft II micromanagement tasks demonstrate that QPLEX significantly outperforms state-of-the-art baselines in both online and offline data collection settings, and also reveal that QPLEX achieves high sample efficiency and can benefit from offline datasets without additional online exploration.

Motivation & Objective

• Motivate scalable value-based multi-agent reinforcement learning under CTDE with the IGM principle.
• Propose a duplex dueling network architecture to factorize the joint value function while preserving exact IGM consistency.
• Formalize advantage-based IGM and show its equivalence to IGM, enabling complete function expressiveness.
• Demonstrate superior performance and sample efficiency in online and offline STARCRAFT II micromanagement benchmarks.

Proposed method

• Introduce advantage-based IGM to convert the IGM constraint into bounds on the advantage functions.
• Use a duplex dueling architecture that combines individually learned Q-functions into a joint Q-function via a Transformation network and a Dueling Mixing network.
• Adopt a joint advantage computation with positive attention-based weights to satisfy IGM constraints, enabling scalable factorization (A_tot = sum_i lambda_i A_i).
• Implement the QPLEX network with an Individual Action-Value Function per agent and a centralized Transformation module that conditions on joint history.
• Train end-to-end with TD loss and target networks, while execution uses the decentralized individual Q-functions.

Experimental results

Research questions

• RQ1 Can advantage-based IGM provide a complete function class for joint action-value factorization in CTDE MARL?
• RQ2 Does the duplex dueling architecture enable exact IGM consistency and superior scalability compared to existing factorization methods?
• RQ3 How does QPLEX perform in online and offline STARCRAFT II micromanagement benchmarks relative to baselines like QMIX, VDN, QTRAN, and Qatten?
• RQ4 Does QPLEX improve stability and sample efficiency in both simple didactic tasks and complex multi-agent environments?
• RQ5 Can QPLEX leverage offline data to achieve robust performance without additional online exploration?

Key findings

• QPLEX achieves complete IGM expressiveness through the advantage-based formulation and duplex architecture.
• QPLEX consistently outperforms baselines on StarCraft II micromanagement tasks in online data collection, with strong sample efficiency.
• QPLEX maintains stability in offline data collection and outperforms other baselines in offline MARL settings.
• Ablations show the importance of the attention-based credit assignment weights in learning the joint advantage.
• The approach yields improved performance in both didactic problems and complex MARL benchmarks, addressing instability risks of limited expressiveness methods.