[Paper Review] Modelling the Dynamic Joint Policy of Teammates with Attention Multi-agent DDPG

Hangyu Mao, Zhengchao Zhang+2 more|arXiv (Cornell University)|Nov 13, 2018

Reinforcement Learning in Robotics28 references53 citations

TL;DR

ATTention MADDPG extends Deep Deterministic Policy Gradient with a centralized critic and an attention mechanism to explicitly model dynamic joint policies of teammates in cooperative multi-agent tasks, improving scalability and robustness.

ABSTRACT

Modelling and exploiting teammates' policies in cooperative multi-agent systems have long been an interest and also a big challenge for the reinforcement learning (RL) community. The interest lies in the fact that if the agent knows the teammates' policies, it can adjust its own policy accordingly to arrive at proper cooperations; while the challenge is that the agents' policies are changing continuously due to they are learning concurrently, which imposes difficulty to model the dynamic policies of teammates accurately. In this paper, we present \\emph{ATTention Multi-Agent Deep Deterministic Policy Gradient} (ATT-MADDPG) to address this challenge. ATT-MADDPG extends DDPG, a single-agent actor-critic RL method, with two special designs. First, in order to model the teammates' policies, the agent should get access to the observations and actions of teammates. ATT-MADDPG adopts a centralized critic to collect such information. Second, to model the teammates' policies using the collected information in an effective way, ATT-MADDPG enhances the centralized critic with an attention mechanism. This attention mechanism introduces a special structure to explicitly model the dynamic joint policy of teammates, making sure that the collected information can be processed efficiently. We evaluate ATT-MADDPG on both benchmark tasks and the real-world packet routing tasks. Experimental results show that it not only outperforms the state-of-the-art RL-based methods and rule-based methods by a large margin, but also achieves better performance in terms of scalability and robustness.

Motivation & Objective

• Motivate and address the challenge of modeling teammates' dynamically changing policies in cooperative multi-agent systems.
• Propose a scalable, decentralized actor with a centralized attention-based critic to capture joint teammate policies.
• Enable effective coordination in continuous-action multi-agent tasks through adaptive policy modeling.
• Demonstrate improved performance, scalability, and robustness on real-world routing and benchmark tasks.

Proposed method

• Extend DDPG with a centralized critic that accesses observations and actions of all agents.
• Introduce a K-head module to estimate action-conditional Q-values for different teammate actions.
• Apply a Soft Attention mechanism to weigh the action-conditional Q-values into a contextual Q-value.
• Use an attention-weighted aggregation to approximate the teammates’ joint policy distribution without modeling every joint action explicitly.
• Transform the multi-dimensional contextual Q-value into a scalar Q-value for standard TD learning (with a K-head adaptation for continuous actions).
• Train end-to-end under a centralized training, decentralized execution paradigm.

Experimental results

Research questions

• RQ1 How can a centralized critic with attention be used to model dynamic joint policies of teammates in cooperative MARL?
• RQ2 Does ATT-MADDPG improve learning stability, scalability, and performance over state-of-the-art methods in continuous-action multi-agent tasks?
• RQ3 Can a small K-head attention structure effectively group similar teammate actions to approximate joint policy distributions?
• RQ4 What is the impact of the attention mechanism on robustness to changing teammate policies?

Key findings

• ATT-MADDPG outperforms state-of-the-art RL-based and rule-based methods on both real-world packet routing and benchmark tasks.
• The attention mechanism improves scalability and robustness, especially as the number of agents increases.
• The K-head module can group similar teammate actions, enabling efficient approximation of joint policies.
• A combination of the K-head module and attention is necessary for strong performance (K-head alone is insufficient).
• Experiments show better reward outcomes and cooperation compared to MADDPG and PSMADDPGV2 in the tested topologies.