[论文解读] Off-Policy Multi-Agent Decomposed Policy Gradients

Yihan Wang, Beining Han+3 more|arXiv (Cornell University)|Jul 24, 2020

Reinforcement Learning in Robotics参考文献 59被引用 42

一句话总结

DOP introducing a decomposed, off-policy MAPG framework with a linearly factorized critic to enable efficient off-policy learning, mitigate centralized-decentralized mismatch, and improve credit assignment in both discrete and continuous action spaces, showing strong empirical gains on StarCraft II and multi-agent environments.

ABSTRACT

Multi-agent policy gradient (MAPG) methods recently witness vigorous progress. However, there is a significant performance discrepancy between MAPG methods and state-of-the-art multi-agent value-based approaches. In this paper, we investigate causes that hinder the performance of MAPG algorithms and present a multi-agent decomposed policy gradient method (DOP). This method introduces the idea of value function decomposition into the multi-agent actor-critic framework. Based on this idea, DOP supports efficient off-policy learning and addresses the issue of centralized-decentralized mismatch and credit assignment in both discrete and continuous action spaces. We formally show that DOP critics have sufficient representational capability to guarantee convergence. In addition, empirical evaluations on the StarCraft II micromanagement benchmark and multi-agent particle environments demonstrate that DOP significantly outperforms both state-of-the-art value-based and policy-based multi-agent reinforcement learning algorithms. Demonstrative videos are available at https://sites.google.com/view/dop-mapg/.

研究动机与目标

• 诊断为什么最先进的多智能体策略梯度方法在价值基方法上表现不佳。
• 提出一个分解评估器架构，以实现离策略 MAPG 并解决 CDM 与信用分配问题。
• 为分解评估器建立理论收敛性质。
• 在离散（StarCraft II SMAC）和连续（multi-agent particle）基准上展示经验有效性。

提出的方法

• 提出一个集中但分解的评估器：Q_tot^φ(τ, a) = sum_i k_i(τ) Q_i^φ_i(τ, a_i) + b(τ).
• 通过将 E_π[Q_tot^φ(τ, ·)] 减少为一组逐智能体的期望之和，从而实现可处理的离策略评估。
• 将分解的树备份与基于策略的 TD(λ) 更新相结合，以在样本与训练效率之间取得平衡。
• 为离散和连续动作空间提供随机和确定性的 DOP 变体，并给出相应的策略梯度形式。
• 证明分解的评估器在温和假设下保持策略改进，并显示偏差-方差权衡。
• 在附录中描述实现细节和网络架构。

实验结果

研究问题

• RQ1 在实际中是否会出现集中-去中心化不匹配（CDM）？分解的评估器能否减轻它？
• RQ2 能否通过分解评估器使离策略随机 MAPG 方法可控且稳定？
• RQ3 分解的评估器是否能为确定性策略提供有效的多智能体信用分配？
• RQ4 DOP 方法在标准基准上是否优于最先进的 MAPG 和基于价值的 MARL 方法？

主要发现

• 随机 DOP 在离散动作的 StarCraft II 微观管理任务上显著超越基线。
• DOP 通过分解评估器结构降低梯度方差并减轻 CDM。
• 离策略分解树备份 TB 使得多智能体离策略评估成为可控。
• 确定性 DOP 在连续动作任务中学习信用分配，并且优于之前的 MAPG 方法。
• DOP 展现出在多种随机种子下的稳定性与强劲表现。