[Paper Review] Learning Multiagent Communication with Backpropagation

Sainbayar Sukhbaatar, Arthur Szlam+1 more|arXiv (Cornell University)|May 25, 2016

Reinforcement Learning in Robotics37 references276 citations

TL;DR

CommNet enables continuous, differentiable communication among cooperating agents, learned end-to-end with backpropagation, boosting performance on partially observable multi-agent tasks.

ABSTRACT

Many tasks in AI require the collaboration of multiple agents. Typically, the communication protocol between agents is manually specified and not altered during training. In this paper we explore a simple neural model, called CommNet, that uses continuous communication for fully cooperative tasks. The model consists of multiple agents and the communication between them is learned alongside their policy. We apply this model to a diverse set of tasks, demonstrating the ability of the agents to learn to communicate amongst themselves, yielding improved performance over non-communicative agents and baselines. In some cases, it is possible to interpret the language devised by the agents, revealing simple but effective strategies for solving the task at hand.

Motivation & Objective

• Motivate and enable cooperative multi-agent tasks where communication is not pre-specified but learned.
• Introduce a simple neural model (CommNet) that uses continuous communication learned alongside agents’ policies.
• Demonstrate the approach across diverse tasks with partial observability and dynamic agent counts.
• Show interpretability of learned communication patterns in certain tasks.

Proposed method

• Propose CommNet where each agent runs a shared neural module receiving its own state and a broadcasted continuous communication vector.
• Define a propagation scheme with multiple communication steps (K) where each step updates agent hidden states and aggregates others’ states via a normalized sum to form c^i_j.
• Formalize special cases including local connectivity, skip connections, and temporal recurrence (RNN/LSTM).
• Train end-to-end via backpropagation when supervision is available; otherwise use reinforcement learning with policy gradients and baselines as needed.
• Provide baselines: Independent controllers, Fully-connected networks, and Discrete communication methods for comparison.
• Apply the model to tasks including a lever-pulling game, MazeBase-based traffic junction and combat scenarios, and bAbI QA tasks to test versatility.

Experimental results

Research questions

• RQ1 Can agents jointly learn a communication protocol that improves coordination under partial observability?
• RQ2 Does continuous, differentiable communication yield better performance than non-communicative or discretely communicating baselines?
• RQ3 How does the communication mechanism scale with dynamic agent counts and varying visibility?
• RQ4 What interpretable structures or patterns emerge in the learned communication?
• RQ5 How does CommNet perform across diverse multi-agent tasks (traffic, combat, QA) compared to strong baselines?

Key findings

• CommNet outperforms independent controllers and fully-connected baselines across tasks with partial visibility.
• In the lever-pulling task, CommNet achieves near-perfect performance under supervised and strong performance under reinforcement learning.
• In traffic junction experiments, CommNet substantially reduces failure rates across MLP, RNN, and LSTM modules, with local connectivity performing best in hard variants.
• In combat tasks, CommNet consistently improves win rates across varying team sizes and visibility, with LSTM modules often best.
• On bAbI QA tasks, CommNet reduces mean error compared to an independent MLP baseline, though it underperforms MemN2N which is tailored for long-story reasoning.
• Analyses show learned communications are sparse yet meaningful, with distinct communication clusters corresponding to task-relevant events.