[Paper Review] A Joint Learning and Communication Framework for Multi-Agent Reinforcement Learning over Noisy Channels.

Tze-Yang Tung, Joan S. Pujol Roig+2 more|arXiv (Cornell University)|Jan 2, 2021

Reinforcement Learning in Robotics36 references4 citations

TL;DR

This paper proposes a joint learning and communication framework for multi-agent reinforcement learning (MARL) in which agents coordinate via noisy channels as an intrinsic part of the environment dynamics. By integrating communication into the MA-POMDP framework, agents learn to cooperate effectively even over unreliable channels, outperforming separate communication and learning approaches in coordination tasks.

ABSTRACT

We propose a novel formulation of the effectiveness problem in communications, put forth by Shannon and Weaver in their seminal work [2], by considering multiple agents communicating over a noisy channel in order to achieve better coordination and cooperation in a multi-agent reinforcement (MARL) framework. Specifically, we consider a multi-agent partially observable Markov decision process (MA-POMDP), in which the agents, in addition to interacting with the environment can also with each other over a noisy communication channel. The noisy communication channel is considered explicitly as part of the dynamics of the environment and the message each agent sends is part of the action that the agent can take. As a result, the agents learn not only to collaborate with each other but also to effectively over a noisy channel. This framework generalizes both the traditional communication problem, where the main goal is to convey a message reliably over a noisy channel, and the learning to communicate framework that has received recent attention in the MARL literature, where the underlying communication channels are assumed to be error-free. We show via examples that the joint policy learned using the proposed framework is superior to that where the communication is considered separately from the underlying MA-POMDP. This is a very powerful framework, which has many real world applications, from autonomous vehicle planning to drone swarm control, and opens up the rich toolbox of deep reinforcement for the design of multi-user communication systems.

Motivation & Objective

• To address the challenge of effective communication in multi-agent reinforcement learning under noisy channel conditions.
• To unify the traditional communication problem with MARL by modeling noisy channels as part of the environment dynamics.
• To enable agents to jointly learn optimal policies and robust communication strategies in partially observable environments.
• To generalize existing frameworks that assume error-free communication or treat communication as a separate module.

Proposed method

• Formulate the multi-agent partially observable Markov decision process (MA-POMDP) with communication channels explicitly modeled as part of the environment dynamics.
• Treat each agent's message as part of its action space, enabling end-to-end learning of communication and coordination.
• Model the noisy channel as a stochastic transition in the environment, affecting message reliability.
• Use deep reinforcement learning to jointly optimize policies and communication strategies.
• Train agents to adapt messages based on channel conditions, improving robustness and coordination.
• Generalize prior frameworks by integrating communication into the learning process rather than treating it as a separate component.

Experimental results

Research questions

• RQ1 How does joint learning of communication and coordination over noisy channels improve multi-agent performance compared to separate learning?
• RQ2 What is the impact of explicit channel modeling on policy robustness and coordination in MA-POMDPs?
• RQ3 Can agents learn effective communication strategies that adapt to channel noise without relying on error-free assumptions?
• RQ4 How does the proposed framework compare to existing MARL and communication-only approaches in terms of coordination quality?
• RQ5 What are the implications of this framework for real-world applications such as drone swarms or autonomous vehicles?

Key findings

• The joint learning framework significantly improves coordination performance compared to approaches that separate communication from learning.
• Agents trained with the proposed framework achieve better task success rates in multi-agent coordination tasks under noisy conditions.
• The framework enables agents to learn communication strategies that are robust to channel errors, reducing reliance on perfect transmission.
• The integration of noisy channels into the MA-POMDP dynamics leads to more realistic and effective policy learning.
• The approach generalizes prior work by unifying communication and learning, demonstrating superior performance in complex, noisy environments.
• The framework is applicable to real-world systems such as autonomous vehicle coordination and drone swarm control, where communication reliability is critical.