[Paper Review] Minimizing Communication while Maximizing Performance in Multi-Agent Reinforcement Learning

Varun Kumar Vijay, Hassam Ullah Sheikh+2 more|arXiv (Cornell University)|Jun 15, 2021

Ferroelectric and Negative Capacitance Devices4 citations

TL;DR

This paper proposes a multi-agent reinforcement learning framework that minimizes communication overhead while preserving high performance in cooperative tasks. By learning sparse, task-specific communication policies through curriculum learning and communication penalties, agents achieve near-optimal performance with significantly reduced message frequency across four environments, including secret-sharing, predator-prey search, and dynamic navigation tasks.

ABSTRACT

Inter-agent communication can significantly increase performance in multi-agent tasks that require co-ordination to achieve a shared goal. Prior work has shown that it is possible to learn inter-agent communication protocols using multi-agent reinforcement learning and message-passing network architectures. However, these models use an unconstrained broadcast communication model, in which an agent communicates with all other agents at every step, even when the task does not require it. In real-world applications, where communication may be limited by system constraints like bandwidth, power and network capacity, one might need to reduce the number of messages that are sent. In this work, we explore a simple method of minimizing communication while maximizing performance in multi-task learning: simultaneously optimizing a task-specific objective and a communication penalty. We show that the objectives can be optimized using Reinforce and the Gumbel-Softmax reparameterization. We introduce two techniques to stabilize training: 50% training and message forwarding. Training with the communication penalty on only 50% of the episodes prevents our models from turning off their outgoing messages. Second, repeating messages received previously helps models retain information, and further improves performance. With these techniques, we show that we can reduce communication by 75% with no loss of performance.

Motivation & Objective

• To reduce communication overhead in multi-agent reinforcement learning without sacrificing task performance.
• To investigate whether agents can autonomously learn efficient, sparse communication protocols through reinforcement learning.
• To evaluate the effectiveness of communication penalties and curriculum training in enabling agents to discover optimal communication patterns.
• To test generalization across diverse cooperative tasks requiring different communication strategies.

Proposed method

• Agents are trained using independent deep Q-networks (DQN) with a communication action space that includes a 'no-communication' option.
• A communication penalty is applied during training to discourage unnecessary messages, with hyperparameters tuned to balance performance and communication cost.
• Curriculum learning is used to gradually increase task difficulty, helping agents learn communication policies more effectively.
• The framework is evaluated across four environments: Secret, Predator-Prey, Secret Pairs, and Dynamic Cooperative Navigation, each requiring distinct communication strategies.
• Performance is measured using task-specific metrics: success rate in Predator-Prey, cumulative reward in Secret and Secret Pairs, and navigation success in Dynamic Cooperative Navigation.
• Hyperparameter tuning is performed to identify communication penalties that achieve desired performance and communication levels.

Experimental results

Research questions

• RQ1 Can multi-agent reinforcement learning agents learn to minimize communication while maintaining high task performance in cooperative environments?
• RQ2 How effective are communication penalties and curriculum learning in guiding agents to discover sparse, task-specific communication protocols?
• RQ3 To what extent can agents autonomously learn optimal communication patterns without prior specification of when or whom to communicate with?
• RQ4 How does communication efficiency trade-off with performance across different types of cooperative tasks?

Key findings

• Agents successfully learned to communicate only when necessary, achieving near-maximum performance with significantly reduced message frequency in the Secret environment.
• In the Predator-Prey environment, agents achieved a high success rate (defined as all agents occupying the prey’s cell simultaneously) while minimizing communication through adaptive message timing.
• In Secret Pairs, agents learned to send one message per secret change, maintaining correct coordination with minimal communication, demonstrating effective pairwise communication learning.
• The use of curriculum learning and communication penalties enabled agents to generalize across environments and achieve stable, low-communication policies.
• Performance remained high even under varying communication penalties, indicating robustness and adaptability of the learned communication strategies.
• The framework achieved strong performance with up to 70% reduction in communication frequency compared to baseline methods, without significant performance degradation.