[Paper Review] Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks

Jakob Foerster, Yannis Assael+2 more|arXiv (Cornell University)|Feb 8, 2016

Reinforcement Learning in Robotics43 references85 citations

TL;DR

This paper introduces Deep Distributed Recurrent Q-Networks (DDRQN), a multi-agent reinforcement learning framework that enables agents to autonomously learn communication protocols for solving partially observable coordination tasks. In two riddle-based environments—hats and switch riddles—DDRQN successfully discovers effective communication strategies, marking the first successful deep reinforcement learning approach to learning communication protocols from scratch.

ABSTRACT

We propose deep distributed recurrent Q-networks (DDRQN), which enable teams of agents to learn to solve communication-based coordination tasks. In these tasks, the agents are not given any pre-designed communication protocol. Therefore, in order to successfully communicate, they must first automatically develop and agree upon their own communication protocol. We present empirical results on two multi-agent learning problems based on well-known riddles, demonstrating that DDRQN can successfully solve such tasks and discover elegant communication protocols to do so. To our knowledge, this is the first time deep reinforcement learning has succeeded in learning communication protocols. In addition, we present ablation experiments that confirm that each of the main components of the DDRQN architecture are critical to its success.

Motivation & Objective

• Address the challenge of multi-agent reinforcement learning in partially observable environments where agents must coordinate without pre-defined communication protocols.
• Develop a scalable deep learning architecture that allows agents to learn coordinated policies and emergent communication strategies.
• Demonstrate that end-to-end deep reinforcement learning can discover effective, human-interpretable communication protocols in complex coordination tasks.
• Investigate the necessity and impact of key architectural components in multi-agent, partially observable settings.
• Provide a benchmark for evaluating communication-based coordination in multi-agent RL using well-known riddles as testbeds.

Proposed method

• Propose DDRQN, a deep reinforcement learning architecture combining recurrent neural networks (LSTMs) with shared, agent-specific weights to model memory and history.
• Introduce last-action inputs to help agents approximate their own action-observation history, enabling better sequence modeling.
• Implement inter-agent weight sharing with agent-specific conditioning via unique IDs, enabling fast, generalizable learning across agents.
• Disable experience replay to avoid non-stationarity issues arising from concurrent learning by multiple agents.
• Train agents using independent Q-learning with deep Q-networks, where each agent learns its own Q-function based on local observations and actions.
• Use a loss function minimizing the temporal difference error between target and predicted Q-values, with target networks for stability.

Experimental results

Research questions

• RQ1 Can deep reinforcement learning learn effective communication protocols in multi-agent, partially observable environments without any pre-defined communication rules?
• RQ2 What architectural components are essential for enabling successful communication and coordination in such settings?
• RQ3 How does the absence of experience replay affect learning stability and performance in multi-agent, non-stationary environments?
• RQ4 To what extent can shared, recurrent networks with agent-specific conditioning enable generalization and fast learning across agents?
• RQ5 Can emergent communication protocols discovered by DDRQN be both effective and interpretable in complex coordination tasks?

Key findings

• DDRQN successfully solves both the hats and switch riddles with high success rates, achieving near-perfect performance on the switch riddle for up to n=4 agents.
• Inter-agent weight sharing is the most critical component; without it, agents fail to learn the task even for n=3.
• Last-action inputs significantly improve performance, as removing them reduces success to levels comparable to a naive 'tell on last day' strategy.
• Disabling experience replay is essential—performance with replay never reaches optimal, indicating that replay is poorly suited to the non-stationarity of multi-agent learning.
• The ablation study confirms that all three core components (last-action inputs, weight sharing, no experience replay) are necessary for effective learning.
• DDRQN discovers interpretable, elegant communication protocols that align with human solutions to the riddles, demonstrating emergent coordination without supervision.