[Paper Review] R-MADDPG for Partially Observable Environments and Limited Communication

Rose E. Wang, Michael Everett+1 more|arXiv (Cornell University)|Feb 16, 2020

Reinforcement Learning in RoboticsComputer Science25 references64 citations

TL;DR

Introduces R-MADDPG, a recurrent multiagent actor-critic framework for coordination under partial observability and limited communication, showing the recurrent critic is crucial for learning in real-world-like MARL tasks.

ABSTRACT

There are several real-world tasks that would benefit from applying multiagent reinforcement learning (MARL) algorithms, including the coordination among self-driving cars. The real world has challenging conditions for multiagent learning systems, such as its partial observable and nonstationary nature. Moreover, if agents must share a limited resource (e.g. network bandwidth) they must all learn how to coordinate resource use. This paper introduces a deep recurrent multiagent actor-critic framework (R-MADDPG) for handling multiagent coordination under partial observable set-tings and limited communication. We investigate recurrency effects on performance and communication use of a team of agents. We demonstrate that the resulting framework learns time dependencies for sharing missing observations, handling resource limitations, and developing different communication patterns among agents.

Motivation & Objective

• Address partial observability, nonstationarity, and limited inter-agent communication in real-world MARL settings.
• Develop a recurrent multiagent actor-critic model that jointly learns movement and communication policies.
• Demonstrate the importance of a recurrent critic for learning under partial observability and communication constraints.
• Provide an open-source implementation of R-MADDPG for replication and extension.

Proposed method

• Extend MADDPG to a fully recurrent actor-critic architecture for multiagent coordination.
• Learn two policies in parallel: one for physical navigation and one for communication.
• Use three recurrent model variants to study the role of recurrence in actors and critics.
• Train with centralized critic including all agents’ observations and actions to mitigate nonstationarity.
• Evaluate under partial observability and limited communication budgets to analyze performance and emergent communication patterns.
• Provide an open-source implementation at the referenced GitHub repository.

Experimental results

Research questions

• RQ1 Can recurrent architectures enable effective coordination under partial observability and limited communication?
• RQ2 Is a recurrent critic essential for learning in partially observable MARL settings, and how does it compare to recurrent actors alone?
• RQ3 How does communication budget influence coordination performance and emergent strategies?
• RQ4 What are the emergent communication and coordination patterns when bandwidth is constrained?

Key findings

• The fully recurrent actor-critic model enables learning under partial observability and communication limits.
• The recurrent critic is the crucial component that enables learning in partially observable, multiagent environments; a recurrent actor alone is insufficient.
• MADDPG struggles under partial observability and limited communication, highlighting the need for recurrence in the critic.
• Increasing the communication budget improves performance and reduces reward variance, indicating a tradeoff between bandwidth and coordination quality.
• R-MADDPG achieves coordinated arrival at the goal under varying communication budgets, with emergent patterns such as waiting or moving to synchronize arrivals under limited messages.