[论文解读] FACMAC: Factored Multi-Agent Centralised Policy Gradients

Bei Peng, Tabish Rashid+5 more|arXiv (Cornell University)|Mar 14, 2020

Reinforcement Learning in Robotics参考文献 70被引用 106

一句话总结

FACMAC 引入一个集中但因子化的评估者，带有集中梯度估计器，用于合作 MARL，能够实现连续和离散动作任务，并在多个领域超过 MADDPG 和 baselines。

ABSTRACT

We propose FACtored Multi-Agent Centralised policy gradients (FACMAC), a new method for cooperative multi-agent reinforcement learning in both discrete and continuous action spaces. Like MADDPG, a popular multi-agent actor-critic method, our approach uses deep deterministic policy gradients to learn policies. However, FACMAC learns a centralised but factored critic, which combines per-agent utilities into the joint action-value function via a non-linear monotonic function, as in QMIX, a popular multi-agent Q-learning algorithm. However, unlike QMIX, there are no inherent constraints on factoring the critic. We thus also employ a nonmonotonic factorisation and empirically demonstrate that its increased representational capacity allows it to solve some tasks that cannot be solved with monolithic, or monotonically factored critics. In addition, FACMAC uses a centralised policy gradient estimator that optimises over the entire joint action space, rather than optimising over each agent's action space separately as in MADDPG. This allows for more coordinated policy changes and fully reaps the benefits of a centralised critic. We evaluate FACMAC on variants of the multi-agent particle environments, a novel multi-agent MuJoCo benchmark, and a challenging set of StarCraft II micromanagement tasks. Empirical results demonstrate FACMAC's superior performance over MADDPG and other baselines on all three domains.

研究动机与目标

• 激发并开发一个可扩展的集中但因子化的评估者，用于合作多智能体强化学习。
• 使得在完整联合动作空间上的政策优化成为可能，以提升协调。
• 展示非单调因子化和更大表征容量的好处。
• 展示在具有挑战性的任务中对离散和连续动作空间的适用性。

提出的方法

• 定义一个集中但因子化的评估者，通过一个（非）线性混合函数将各智能体效用整合。
• 引入一个集中式梯度估计器，对整个联合动作空间进行优化。
• 探索评估者的单调（QMIX 风格）和非单调因子化。
• 使用 Gumbel-Softmax 配合 Straight-Through 估计器将其适配到离散动作。
• 在连续和离散的 MARL 基准上进行评估，包含 MAMuJoCo、Continuous Predator-Prey 和 SMAC。

实验结果

研究问题

• RQ1 集中但因子化的评估者是否比单一评估者在 MARL 中改善协作？
• RQ2 非单调因子化是否提供更大的表征容量来解决复杂任务？
• RQ3 与每个智能体梯度相比，集中式策略梯度估计是否有利于学习？
• RQ4 FACMAC 在连续 vs 离散动作域上的表现如何，并且在更多智能体时的可扩展性如何？

主要发现

• FACMAC 在连续与离散的协作任务上优于 MADDPG 和其他基线。
• 将评估者因子化使得随着智能体/动作数量增长的可扩展性更好。
• 非单调因子化能解决单调或单一评估者无法解决的任务。
• 集中式梯度估计提升协作并有助于在简单和复杂任务中避免局部最优。
• FACMAC 能扩展到更大量的智能体和复杂域，如 MAMuJoCo 和 SMAC，在多张地图上表现优于基线。