[Paper Review] From Explicit Communication to Tacit Cooperation:A Novel Paradigm for Cooperative MARL

Dapeng Li, Zhiwei Xu+2 more|arXiv (Cornell University)|Apr 28, 2023

Distributed Control Multi-Agent Systems4 citations

TL;DR

This paper proposes TACO, a novel cooperative MARL paradigm that transitions from explicit communication to tacit cooperation by combining real-time global information with trajectory-reconstructed global features during training. By progressively reducing reliance on explicit communication, TACO achieves performance comparable to or better than communication-based methods like QMIX, even under fully decentralized execution without any communication in inference.

ABSTRACT

Centralized training with decentralized execution (CTDE) is a widely-used learning paradigm that has achieved significant success in complex tasks. However, partial observability issues and the absence of effectively shared signals between agents often limit its effectiveness in fostering cooperation. While communication can address this challenge, it simultaneously reduces the algorithm's practicality. Drawing inspiration from human team cooperative learning, we propose a novel paradigm that facilitates a gradual shift from explicit communication to tacit cooperation. In the initial training stage, we promote cooperation by sharing relevant information among agents and concurrently reconstructing this information using each agent's local trajectory. We then combine the explicitly communicated information with the reconstructed information to obtain mixed information. Throughout the training process, we progressively reduce the proportion of explicitly communicated information, facilitating a seamless transition to fully decentralized execution without communication. Experimental results in various scenarios demonstrate that the performance of our method without communication can approaches or even surpasses that of QMIX and communication-based methods.

Motivation & Objective

• To address the limitations of centralized training with decentralized execution (CTDE) in partially observable multi-agent environments where lack of shared signals hinders cooperation.
• To overcome the practicality issues of communication-based MARL methods due to high communication costs and deployment constraints.
• To enable effective cooperation in decentralized execution by reconstructing global information from agents' local trajectories.
• To develop a training paradigm that smoothly transitions from explicit communication to tacit cooperation, minimizing performance drop during communication-free inference.
• To demonstrate that reconstructed global information can effectively support cooperation without requiring real-time communication.

Proposed method

• Introduce a mixed information representation that combines explicitly communicated global information and reconstructed global features from each agent’s local trajectory.
• Use an attention mechanism to dynamically weight and fuse reconstructed and true global information during training.
• Gradually reduce the contribution of explicit communication (via a decayed coefficient α) over training, enabling a progressive shift to tacit cooperation.
• Train the policy using a hybrid loss that incorporates both value-based learning (e.g., QMIX) and reconstruction loss to improve the accuracy of trajectory-based global feature reconstruction.
• Leverage t-SNE visualization to validate that reconstructed features closely match the distribution of true global features.
• Apply the TACO framework to existing CTDE algorithms like VDN and QMIX, creating TACO-VDN and TACO-QMIX.

Experimental results

Research questions

• RQ1 Can a training paradigm that transitions from explicit communication to tacit cooperation achieve effective multi-agent cooperation without communication during inference?
• RQ2 How well can agents reconstruct global information from their own local trajectories, and does this reconstruction support effective cooperation?
• RQ3 Does progressively reducing communication during training improve the performance of communication-free execution?
• RQ4 Can TACO outperform both standard CTDE methods and communication-based MARL algorithms in complex, partially observable environments?
• RQ5 What hyperparameters (e.g., attention dimension, communication decay rate) most significantly affect the performance of the TACO framework?

Key findings

• TACO-VDN and TACO-QMIX outperform their respective baseline models (VDN and QMIX) in all evaluated scenarios, including challenging environments like academy_counterattack_hard in GRF.
• In the GRF environment, TACO-QMIX achieves superior performance compared to QMIX-Attention, demonstrating that communication-free execution with reconstruction can surpass communication-based methods.
• Visualization using t-SNE shows that reconstructed global features closely follow the distribution of true global features, indicating that reconstruction is meaningful and informative.
• Ablation studies confirm that a progressive decay of communication (α) leads to better performance than immediate full communication (TACO-Leap), highlighting the importance of the gradual transition.
• Optimal performance is achieved with lower attention dimensions and moderate β values, suggesting that simpler reconstruction heads are more effective for trajectory-based global feature learning.
• The method achieves state-of-the-art performance in communication-free execution, matching or exceeding the performance of communication-based methods like QMIX-Attention in complex tasks.