[论文解读] A Joint Learning and Communication Framework for Multi-Agent Reinforcement Learning over Noisy Channels.

Tze-Yang Tung, Joan S. Pujol Roig+2 more|arXiv (Cornell University)|Jan 2, 2021

Reinforcement Learning in Robotics参考文献 36被引用 4

一句话总结

本文提出了一种多智能体强化学习（MARL）的联合学习与通信框架，其中智能体通过噪声信道进行协调，作为环境动态的内在组成部分。通过将通信整合到MA-POMDP框架中，智能体即使在不可靠信道上也能有效合作，其在协调任务中的表现优于将通信与学习分开处理的方法。

ABSTRACT

We propose a novel formulation of the effectiveness problem in communications, put forth by Shannon and Weaver in their seminal work [2], by considering multiple agents communicating over a noisy channel in order to achieve better coordination and cooperation in a multi-agent reinforcement (MARL) framework. Specifically, we consider a multi-agent partially observable Markov decision process (MA-POMDP), in which the agents, in addition to interacting with the environment can also with each other over a noisy communication channel. The noisy communication channel is considered explicitly as part of the dynamics of the environment and the message each agent sends is part of the action that the agent can take. As a result, the agents learn not only to collaborate with each other but also to effectively over a noisy channel. This framework generalizes both the traditional communication problem, where the main goal is to convey a message reliably over a noisy channel, and the learning to communicate framework that has received recent attention in the MARL literature, where the underlying communication channels are assumed to be error-free. We show via examples that the joint policy learned using the proposed framework is superior to that where the communication is considered separately from the underlying MA-POMDP. This is a very powerful framework, which has many real world applications, from autonomous vehicle planning to drone swarm control, and opens up the rich toolbox of deep reinforcement for the design of multi-user communication systems.

研究动机与目标

• 解决在噪声信道条件下多智能体强化学习中有效通信的挑战。
• 通过将噪声信道建模为环境动态的一部分，将传统通信问题与MARL统一起来。
• 使智能体能够联合学习最优策略与鲁棒的通信策略，适用于部分可观察环境。
• 推广现有框架，这些框架假设通信无误或将通信视为独立模块。

提出的方法

• 将多智能体部分可观察马尔可夫决策过程（MA-POMDP）中的通信信道显式建模为环境动态的一部分。
• 将每个智能体的消息视为其动作空间的一部分，从而实现通信与协调的端到端学习。
• 将噪声信道建模为环境中的随机转移，影响消息的可靠性。
• 使用深度强化学习联合优化策略与通信策略。
• 训练智能体根据信道条件自适应调整消息，提升鲁棒性与协调性。
• 通过将通信整合到学习过程中而非将其视为独立组件，推广了先前的框架。

实验结果

研究问题

• RQ1 与分开学习通信和协调的方法相比，联合学习噪声信道上的通信与协调在多智能体性能上有哪些提升？
• RQ2 显式建模信道对MA-POMDP中策略鲁棒性与协调性有何影响？
• RQ3 智能体能否在不依赖无误传输假设的前提下，学习到适应信道噪声的有效通信策略？
• RQ4 与现有的MARL和仅通信方法相比，该框架在协调质量方面表现如何？
• RQ5 该框架对现实世界应用（如无人机编队或自动驾驶汽车）有何启示？

主要发现

• 联合学习框架显著提升了与将通信与学习分离的方法相比的协调性能。
• 在噪声条件下，采用该框架训练的智能体在多智能体协调任务中取得了更高的任务成功率。
• 该框架使智能体能够学习对信道错误具有鲁棒性的通信策略，减少了对完美传输的依赖。
• 将噪声信道整合到MA-POMDP动态中，促进了更真实且高效的策略学习。
• 该方法通过统一通信与学习，推广了先前工作，在复杂且噪声较大的环境中表现出更优性能。
• 该框架适用于自动驾驶汽车协调与无人机编队控制等现实世界系统，其中通信可靠性至关重要。