[Paper Review] RGMComm: Return Gap Minimization via Discrete Communications in Multi-Agent Reinforcement Learning

Jingdi Chen, Lü Tian+1 more|arXiv (Cornell University)|Aug 7, 2023

Reinforcement Learning in Robotics4 citations

TL;DR

RGMComm proposes a discrete communication framework for multi-agent reinforcement learning that minimizes the return gap between partially observable policies and an ideal fully observable policy. By modeling message generation as online clustering using a cosine-distance-based Regularized Information Maximization loss, RGMComm achieves state-of-the-art performance with interpretable, few-bit messages and provides a closed-form upper bound on the return gap, ensuring theoretical performance guarantees.

ABSTRACT

Communication is crucial for solving cooperative Multi-Agent Reinforcement Learning tasks in partially observable Markov Decision Processes. Existing works often rely on black-box methods to encode local information/features into messages shared with other agents, leading to the generation of continuous messages with high communication overhead and poor interpretability. Prior attempts at discrete communication methods generate one-hot vectors trained as part of agents' actions and use the Gumbel softmax operation for calculating message gradients, which are all heuristic designs that do not provide any quantitative guarantees on the expected return. This paper establishes an upper bound on the return gap between an ideal policy with full observability and an optimal partially observable policy with discrete communication. This result enables us to recast multi-agent communication into a novel online clustering problem over the local observations at each agent, with messages as cluster labels and the upper bound on the return gap as clustering loss. To minimize the return gap, we propose the Return-Gap-Minimization Communication (RGMComm) algorithm, which is a surprisingly simple design of discrete message generation functions and is integrated with reinforcement learning through the utilization of a novel Regularized Information Maximization loss function, which incorporates cosine-distance as the clustering metric. Evaluations show that RGMComm significantly outperforms state-of-the-art multi-agent communication baselines and can achieve nearly optimal returns with few-bit messages that are naturally interpretable.

Motivation & Objective

• To address the lack of theoretical guarantees in existing discrete communication methods for multi-agent reinforcement learning.
• To reduce communication overhead and improve interpretability by replacing continuous, black-box messages with discrete, finite-alphabet messages.
• To minimize the return gap between a partially observable policy with communication and an ideal fully observable policy.
• To enable dynamic, decentralized message generation without pre-agreed semantics, using online clustering of local observations.
• To provide a closed-form upper bound on the return gap that guides message design and training.

Proposed method

• Formalizes the return gap between a fully observable policy and a partially observable policy with discrete communication as an upper bound involving the average cosine-distance between joint action-value vectors in the same cluster.
• Reframes message generation as an online clustering problem, where messages correspond to cluster labels and the return gap upper bound serves as the clustering loss.
• Introduces a novel Regularized Information Maximization (RIM) loss function that incorporates cosine-distance as the clustering metric to minimize the return gap.
• Uses a two-stream neural network architecture with shared weights for action-value estimation and message generation, with normalization via Tanh and adaptive scaling.
• Employs a soft update mechanism for target networks with τ = 0.01 and uses separate learning rates for actor (0.0001) and critic (0.001) networks.
• Applies Gumbel-Softmax relaxation for differentiable message sampling, enabling end-to-end training while maintaining discrete message semantics.

Experimental results

Research questions

• RQ1 Can we derive a closed-form upper bound on the return gap between a fully observable policy and a partially observable policy with discrete communication?
• RQ2 How can we design a discrete communication mechanism that minimizes this return gap while ensuring interpretability and low communication cost?
• RQ3 Can we model message generation as an online clustering problem over local observations, with the return gap as the clustering objective?
• RQ4 Does using cosine-distance as a clustering metric lead to better performance and more interpretable messages than alternative distance measures?
• RQ5 How does RGMComm compare to existing baselines in terms of return, communication efficiency, and interpretability under varying message alphabet sizes?

Key findings

• RGMComm achieves nearly optimal returns with only a few bits per message, significantly outperforming state-of-the-art baselines such as CommNet, IC3Net, TarMAC, and SARNet.
• The method provides a theoretical upper bound on the return gap that scales as O(√ε n Q_max), where ε is the average cosine-distance between action-value vectors in the same cluster.
• Abllation studies confirm that cosine-distance is critical for performance, as alternative normalizations like softmax degrade results.
• RGMComm enables interpretable communication: messages correspond to meaningful clusters of observations, as shown in visualization examples.
• The algorithm learns message semantics dynamically during training without requiring pre-agreed communication protocols or shared vocabularies.
• In grid-world maze environments, RGMComm achieves higher mean episode rewards than centralized Q-learning in some settings, demonstrating that a simple, learnable policy with discrete communication can outperform complex fully observable policies due to reduced combinatorial complexity.