[论文解读] Modelling the Dynamic Joint Policy of Teammates with Attention Multi-agent DDPG

Hangyu Mao, Zhengchao Zhang+2 more|arXiv (Cornell University)|Nov 13, 2018

Reinforcement Learning in Robotics参考文献 28被引用 53

一句话总结

ATT-MADDPG 在 Deep Deterministic Policy Gradient 的基础上扩展了一个集中式评估器和注意力机制，明确建模合作多智能体任务中队友的动态联政策，从而提升可扩展性和鲁棒性。

ABSTRACT

Modelling and exploiting teammates' policies in cooperative multi-agent systems have long been an interest and also a big challenge for the reinforcement learning (RL) community. The interest lies in the fact that if the agent knows the teammates' policies, it can adjust its own policy accordingly to arrive at proper cooperations; while the challenge is that the agents' policies are changing continuously due to they are learning concurrently, which imposes difficulty to model the dynamic policies of teammates accurately. In this paper, we present \\emph{ATTention Multi-Agent Deep Deterministic Policy Gradient} (ATT-MADDPG) to address this challenge. ATT-MADDPG extends DDPG, a single-agent actor-critic RL method, with two special designs. First, in order to model the teammates' policies, the agent should get access to the observations and actions of teammates. ATT-MADDPG adopts a centralized critic to collect such information. Second, to model the teammates' policies using the collected information in an effective way, ATT-MADDPG enhances the centralized critic with an attention mechanism. This attention mechanism introduces a special structure to explicitly model the dynamic joint policy of teammates, making sure that the collected information can be processed efficiently. We evaluate ATT-MADDPG on both benchmark tasks and the real-world packet routing tasks. Experimental results show that it not only outperforms the state-of-the-art RL-based methods and rule-based methods by a large margin, but also achieves better performance in terms of scalability and robustness.

研究动机与目标

• 激发并解决在协作多智能体系统中对队友动态变化策略进行建模的挑战。
• 提出一个可扩展的去中心化行为者，配备一个集中式的基于注意力的评估器，以捕获联合队友策略。
• 通过自适应策略建模，在连续动作的多智能体任务中实现有效协同。
• 在真实路由任务和基准任务中展示性能、可扩展性和鲁棒性的提升。

提出的方法

• 在 DDPG 的基础上扩展一个能够获取所有智能体观测和动作的集中式评估器。
• 引入一个 K-head 模块，用于估计不同队友动作的动作条件Q值。
• 应用 Soft Attention 机制将动作条件Q值加权成上下文Q值。
• 使用带注意力权重的聚合来近似队友联合策略分布，而无需显式建模每个联合动作。
• 将多维上下文Q值转换为标量Q值，以实现标准时序差分学习（对于连续动作有 K-head 自适应）。
• 在集中训练、去中心化执行的范式下端到端训练。

实验结果

研究问题

• RQ1 如何使用带注意力的集中式评估器来建模合作多智能体强化学习中队友的动态联合策略？
• RQ2 在连续动作多智能体任务中，ATT-MADDPG 是否在学习稳定性、可扩展性和性能方面优于最先进的方法？
• RQ3 一个小型的 K-head 注意力结构是否能够有效地将相似的队友动作分组，以近似联合策略分布？
• RQ4 注意力机制对对变化队友策略的鲁棒性有何影响？

主要发现

• ATT-MADDPG 在真实世界的分组路由和基准任务上均优于最先进的基于强化学习和基于规则的方法。
• 注意力机制提升了可扩展性和鲁棒性，尤其是在智能体数量增加时。
• K-head 模块能够将相似的队友动作分组，从而实现对联合策略的高效近似。
• 为了获得强性能，K-head 模块和注意力的结合是必要的（仅有 K-head 不足）。
• 实验在测试拓扑中显示出比 MADDPG 和 PSMADDPGV2 更好的奖励结果和协作。