[Paper Review] Learning to Communicate with Deep Multi-Agent Reinforcement Learning

Jakob Foerster, Yannis Assael+2 more|arXiv (Cornell University)|May 21, 2016

Adversarial Robustness in Machine Learning24 references867 citations

TL;DR

The paper introduces two methods, RIAL and DIAL, for learning communication protocols among cooperative, partially observable agents using centralized learning and deep networks.

ABSTRACT

We consider the problem of multiple agents sensing and acting in environments with the goal of maximising their shared utility. In these environments, agents must learn communication protocols in order to share information that is needed to solve the tasks. By embracing deep neural networks, we are able to demonstrate end-to-end learning of protocols in complex environments inspired by communication riddles and multi-agent computer vision problems with partial observability. We propose two approaches for learning in these domains: Reinforced Inter-Agent Learning (RIAL) and Differentiable Inter-Agent Learning (DIAL). The former uses deep Q-learning, while the latter exploits the fact that, during learning, agents can backpropagate error derivatives through (noisy) communication channels. Hence, this approach uses centralised learning but decentralised execution. Our experiments introduce new environments for studying the learning of communication protocols and present a set of engineering innovations that are essential for success in these domains.

Motivation & Objective

• Investigate how multiple cooperative agents can learn to communicate to maximize shared rewards under partial observability.
• Develop end-to-end learning approaches that enable communication protocols to emerge in deep neural networks.
• Assess the benefits of centralized learning with decentralized execution for training communication-enabled policies.

Proposed method

• Propose Reinforced Inter-Agent Learning (RIAL) which uses deep Q-learning with recurrent networks for environment actions and discrete communication actions.
• Propose Differentiable Inter-Agent Learning (DIAL) which allows real-valued messages to pass between agents during centralized learning and backpropagate through the communication channel.
• Use parameter sharing to enable centralized learning while preserving decentralized execution.
• Discretize real-valued messages during decentralized execution to fit the task’s communication constraints.
• Address partial observability by incorporating recurrent networks and episode-level training dynamics.
• Evaluate with two benchmark multi-agent tasks including Switch Riddle and MNIST-based games.

Experimental results

Research questions

• RQ1 Can agents learn effective communication protocols to solve cooperative tasks under partial observability?
• RQ2 Does differentiable inter-agent communication (DIAL) provide a learning advantage over independent or non-differentiable methods (RIAL) in multi-agent settings?
• RQ3 How do centralised learning, parameter sharing, and channel discretisation affect the emergence of communication?
• RQ4 What kinds of emergent communication protocols arise in complex tasks and how interpretable are they?

Key findings

• RIAL and DIAL can solve the proposed benchmark tasks with centralised learning and distributed execution.
• DIAL with parameter sharing outperforms alternatives and learns protocols faster than RIAL in several tasks.
• Differentiable communication yields richer feedback and leads to more effective message design and coordination than non-differentiable methods.
• Parameter sharing is crucial for learning to communicate in settings with multiple agents.
• DIAL enables the emergence of interpretable, discrete-like communication schemes from continuous protocols under learning.
• The presence of channel noise and regularisation via DRU impacts the learned communication strategies and training dynamics.