[Paper Review] Fully Independent Communication in Multi-Agent Reinforcement Learning

Rafael Pina, De Silva+2 more|arXiv (Cornell University)|Jan 26, 2024

Evolutionary Game Theory and Cooperation4 citations

TL;DR

This paper proposes a novel learning scheme enabling fully independent multi-agent reinforcement learning (MARL) agents—without shared parameters—to successfully learn and utilize communication strategies via distinct policy and message networks. Despite challenges in parameter independence, the method enables effective communication, and results show that communication is not always beneficial and may introduce overhead, especially when agent network capacity is insufficient or excessive.

ABSTRACT

Multi-Agent Reinforcement Learning (MARL) comprises a broad area of research within the field of multi-agent systems. Several recent works have focused specifically on the study of communication approaches in MARL. While multiple communication methods have been proposed, these might still be too complex and not easily transferable to more practical contexts. One of the reasons for that is due to the use of the famous parameter sharing trick. In this paper, we investigate how independent learners in MARL that do not share parameters can communicate. We demonstrate that this setting might incur into some problems, to which we propose a new learning scheme as a solution. Our results show that, despite the challenges, independent agents can still learn communication strategies following our method. Additionally, we use this method to investigate how communication in MARL is affected by different network capacities, both for sharing and not sharing parameters. We observe that communication may not always be needed and that the chosen agent network sizes need to be considered when used together with communication in order to achieve efficient learning.

Motivation & Objective

• Investigate whether effective communication is possible in MARL when agents are fully independent and do not share network parameters.
• Address the challenge of communication in decentralized, non-shared-parameter MARL settings, which are common in practical applications.
• Examine the trade-offs between communication and increased agent network capacity in MARL.
• Evaluate whether communication always improves learning or can introduce unnecessary overhead.
• Propose and validate a new learning scheme that enables communication in fully independent MARL agents

Proposed method

• Introduce a new learning scheme that enables independent agents to learn communication strategies without sharing policy network parameters.
• Equip each agent with a distinct policy network and a separate message generation network, both trained independently.
• Use a communication protocol where agents broadcast messages to others, which are then used as additional input to their policy networks.
• Train agents using a centralized training with decentralized execution (CTDE) framework, where the joint action-value function is optimized using a mixer network.
• Apply a communication loss term to encourage informative message transmission while preserving the independence of the agents’ networks.
• Conduct ablation studies to compare performance across different network capacities and with/without communication

Experimental results

Research questions

• RQ1 Can fully independent MARL agents learn effective communication strategies when they do not share parameters?
• RQ2 How does communication performance vary with different agent network capacities in the absence of parameter sharing?
• RQ3 Does adding communication always improve learning performance, or can it introduce overhead without benefit?
• RQ4 To what extent can increasing agent network capacity compensate for the lack of communication in MARL?
• RQ5 How does the proposed learning scheme enable communication in independent agents without relying on parameter sharing?

Key findings

• Despite the absence of parameter sharing, the proposed learning scheme enables independent MARL agents to successfully learn and utilize communication strategies.
• Communication is not always beneficial and can introduce performance overhead when agent network capacity is too low or too high.
• Higher network capacity can partially compensate for the lack of communication, but only up to a point—excessive capacity without communication leads to diminishing returns.
• The study reveals that communication may be unnecessary in some environments, especially when agents have sufficient individual capacity to process information.
• The results show that the choice of agent network size must be carefully aligned with communication use to achieve efficient learning.
• The proposed method demonstrates that effective communication is feasible in fully independent MARL, even without shared parameters, challenging the assumption that parameter sharing is essential