[Paper Review] Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning

Yaodong Yang, Jianye Hao+5 more|arXiv (Cornell University)|Feb 10, 2020

Reinforcement Learning in Robotics10 references108 citations

TL;DR

Qatten introduces a general framework that decomposes global multiagent Q-values into agent-wise Q-values using a multi-head attention mixer, enabling tractable decentralized policy optimization and improved performance in cooperative MARL, demonstrated on StarCraft II SMAC benchmarks.

ABSTRACT

In many real-world tasks, multiple agents must learn to coordinate with each other given their private observations and limited communication ability. Deep multiagent reinforcement learning (Deep-MARL) algorithms have shown superior performance in such challenging settings. One representative class of work is multiagent value decomposition, which decomposes the global shared multiagent Q-value $Q_{tot}$ into individual Q-values $Q^{i}$ to guide individuals' behaviors, i.e. VDN imposing an additive formation and QMIX adopting a monotonic assumption using an implicit mixing method. However, most of the previous efforts impose certain assumptions between $Q_{tot}$ and $Q^{i}$ and lack theoretical groundings. Besides, they do not explicitly consider the agent-level impact of individuals to the whole system when transforming individual $Q^{i}$s into $Q_{tot}$. In this paper, we theoretically derive a general formula of $Q_{tot}$ in terms of $Q^{i}$, based on which we can naturally implement a multi-head attention formation to approximate $Q_{tot}$, resulting in not only a refined representation of $Q_{tot}$ with an agent-level attention mechanism, but also a tractable maximization algorithm of decentralized policies. Extensive experiments demonstrate that our method outperforms state-of-the-art MARL methods on the widely adopted StarCraft benchmark across different scenarios, and attention analysis is further conducted with valuable insights.

Motivation & Objective

• Motivate the challenge of coordinating multiple agents under partial observations and limited communication.
• Provide a theoretical decomposition of the global Q-value Q_tot in terms of individual Q-values Q^i.
• Propose a practical multi-head attention based mixing network (Qatten) to approximate Q_tot.
• Ensure decentralized policies maximize a centralized value function under CTDE.
• Demonstrate superior performance on StarCraft II SMAC benchmarks and analyze attention weights.

Proposed method

• Derive a decomposition formula: Q_tot(s, a) ≈ c(s) + sum_h sum_i λ_{i,h}(s) Q^i(s, a^i).
• Use a differentiable key-value memory with multi-head attention to approximate coefficients λ_{i,h}(s) for combining Q^i into Q_tot.
• Incorporate an attention-based mixing network that conditions on global state s and agent features u^i to produce agent-level mixing weights.
• Optionally introduce weighted head Q-values to relax monotonicity bounds and enhance representation capacity.
• Maintain monotonicity to preserve IGM (Individual-Global-Max) property for tractable maximization in off-policy learning.

Experimental results

Research questions

• RQ1 Can Q_tot be decomposed in a principled way from per-agent Q^i to enable effective centralized training with decentralized execution?
• RQ2 Can multi-head attention effectively model agent-level impacts on the global Q-value for cooperative MARL?
• RQ3 Does the proposed Qatten framework outperform existing MARL methods on challenging StarCraft II SMAC benchmarks?
• RQ4 How do attention weights λ_{i,h} reflect agents’ roles and importance during tasks, and can they be interpreted to gain insights?

Key findings

• Qatten achieves strong performance, often outperforming state-of-the-art MARL methods on StarCraft SMAC benchmarks across various scenarios.
• The attention-based mixing network provides refined Q_tot representations by incorporating agent-level impacts at multiple orders via multiple heads.
• The approach maintains monotonicity between Q_tot and Q^i, enabling tractable joint action-value maximization under decentralized execution.
• Ablation studies show weighted head Q-values improve performance in difficult scenarios, indicating benefits from increased nonlinearity."
• Attention analysis reveals how different heads assign weights to agents according to their roles and health during battles.