[Paper Review] FACMAC: Factored Multi-Agent Centralised Policy Gradients

Bei Peng, Tabish Rashid+5 more|arXiv (Cornell University)|Mar 14, 2020

Reinforcement Learning in RoboticsComputer Science70 references106 citations

TL;DR

FACMAC introduces a centralised but factored critic with a centralised gradient estimator for cooperative MARL, enabling continuous and discrete action tasks and outperforming MADDPG and baselines across multiple domains.

ABSTRACT

We propose FACtored Multi-Agent Centralised policy gradients (FACMAC), a new method for cooperative multi-agent reinforcement learning in both discrete and continuous action spaces. Like MADDPG, a popular multi-agent actor-critic method, our approach uses deep deterministic policy gradients to learn policies. However, FACMAC learns a centralised but factored critic, which combines per-agent utilities into the joint action-value function via a non-linear monotonic function, as in QMIX, a popular multi-agent Q-learning algorithm. However, unlike QMIX, there are no inherent constraints on factoring the critic. We thus also employ a nonmonotonic factorisation and empirically demonstrate that its increased representational capacity allows it to solve some tasks that cannot be solved with monolithic, or monotonically factored critics. In addition, FACMAC uses a centralised policy gradient estimator that optimises over the entire joint action space, rather than optimising over each agent's action space separately as in MADDPG. This allows for more coordinated policy changes and fully reaps the benefits of a centralised critic. We evaluate FACMAC on variants of the multi-agent particle environments, a novel multi-agent MuJoCo benchmark, and a challenging set of StarCraft II micromanagement tasks. Empirical results demonstrate FACMAC's superior performance over MADDPG and other baselines on all three domains.

Motivation & Objective

• Motivate and develop a scalable centralised but factored critic for cooperative multi-agent RL.
• Enable policy optimization over the full joint action space to improve coordination.
• Demonstrate the benefits of nonmonotonic factorisations and larger representational capacity.
• Showcase applicability to both discrete and continuous action spaces in challenging tasks.

Proposed method

• Define a centralised but factored critic that combines per-agent utilities via a (non)linear mixing function.
• Introduce a centralised gradient estimator that optimises over the entire joint action space.
• Explore both monotonic (QMIX-style) and nonmonotonic factorisations of the critic.
• Adapt to discrete actions using Gumbel-Softmax with Straight-Through estimator.
• Evaluate on continuous and discrete MARL benchmarks including MAMuJoCo, Continuous Predator-Prey, and SMAC.

Experimental results

Research questions

• RQ1 Does a centralised but factored critic improve coordination over monolithic critics in MARL?
• RQ2 Can nonmonotonic factorisations provide greater representational capacity to solve complex tasks?
• RQ3 Does centralised policy gradient estimation benefit learning compared with per-agent gradients?
• RQ4 How does FACMAC perform in continuous vs discrete action domains and scale with more agents?

Key findings

• FACMAC outperforms MADDPG and other baselines on continuous and discrete cooperative tasks.
• Factoring the critic enables better scalability as the number of agents/actions grows.
• Nonmonotonic factorisation can solve tasks that monotonic or monolithic critics cannot.
• Centralised gradient estimation improves coordination and helps avoid local optima in simple and complex tasks.
• FACMAC scales to larger agent counts and complex domains like MAMuJoCo and SMAC, with stronger performance than baselines on multiple maps.