[Paper Review] Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

Tabish Rashid, Mikayel Samvelyan+4 more|arXiv (Cornell University)|Mar 19, 2020

Reinforcement Learning in Robotics429 citations

TL;DR

QMIX is a value-based method for centralized training with decentralized execution that enforces a monotonic mix of per-agent Q-values to guarantee consistent decentralised policies, evaluated on the StarCraft Multi-Agent Challenge (SMAC).

ABSTRACT

In many real-world settings, a team of agents must coordinate its behaviour while acting in a decentralised fashion. At the same time, it is often possible to train the agents in a centralised fashion where global state information is available and communication constraints are lifted. Learning joint action-values conditioned on extra state information is an attractive way to exploit centralised learning, but the best strategy for then extracting decentralised policies is unclear. Our solution is QMIX, a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. QMIX employs a mixing network that estimates joint action-values as a monotonic combination of per-agent values. We structurally enforce that the joint-action value is monotonic in the per-agent values, through the use of non-negative weights in the mixing network, which guarantees consistency between the centralised and decentralised policies. To evaluate the performance of QMIX, we propose the StarCraft Multi-Agent Challenge (SMAC) as a new benchmark for deep multi-agent reinforcement learning. We evaluate QMIX on a challenging set of SMAC scenarios and show that it significantly outperforms existing multi-agent reinforcement learning methods.

Motivation & Objective

• Motivate learning coordinated policies for teams of agents under decentralised execution.
• Propose a centralised but factored Q-function representation that scales with the number of agents.
• Ensure consistency between centralised and decentralised policies through a monotonic mixing architecture.
• Introduce SMAC as a challenging benchmark for deep multi-agent RL in StarCraft II environments.
• Provide an open-source framework (PyMARL) for evaluating multi-agent RL algorithms.

Proposed method

• Represent each agent's value function Q_a with an agent network.
• Combine agent Q-values with a mixing network to produce Q_tot, enforcing monotonicity by using non-negative weights.
• Condition the mixing network weights on global state via hypernetworks to allow state-dependent mixing.
• Prove that monotonicity guarantees that argmax Q_tot equals the stack of argmax Q_a for each agent.
• Train using off-policy methods with centralised training while maintaining decentralised execution during deployment.

Experimental results

Research questions

• RQ1 Can a monotonic mixing of per-agent values guarantee consistency between centralized joint action selection and decentralized agent actions?
• RQ2 Does a monotonic, hypernetwork-conditioned mixing architecture (QMIX) represent a richer class of centralised Q-functions than simple additive decompositions (VDN) while remaining scalable?
• RQ3 Does QMIX outperform existing methods on challenging multi-agent coordination tasks?
• RQ4 How does conditioning the mixing network on global state via hypernetworks affect performance and cooperation in SMAC?

Key findings

• QMIX outperforms IQL, VDN, and COMA on SMAC scenarios in absolute performance and learning speed.
• Monotonicity of the mixing network is sufficient to ensure decentralised argmax aligns with the centralised argmax.
• Conditioning the mixing network on state with hypernetworks enables richer centralised value representations while maintaining scalable decentralised execution.
• A flexible multi-layer mixing network is necessary to achieve robust performance across heterogeneous tasks in SMAC.
• The framework and benchmark (SMAC) enable standardized evaluation of deep multi-agent RL methods beyond gridworld environments.