[Paper Review] Learning Multi-Agent Communication from Graph Modeling Perspective

Shengchao Hu, Li Shen+2 more|arXiv (Cornell University)|May 14, 2024

Advanced Graph Neural Networks4 citations

TL;DR

This paper proposes CommFormer, a novel multi-agent reinforcement learning framework that treats communication architecture as a learnable graph, enabling end-to-end optimization via continuous relaxation and attention mechanisms. By formulating communication structure learning as a bi-level optimization problem, CommFormer achieves superior performance across diverse cooperative tasks, consistently outperforming manually designed architectures and approaching the upper bound of full communication.

ABSTRACT

In numerous artificial intelligence applications, the collaborative efforts of multiple intelligent agents are imperative for the successful attainment of target objectives. To enhance coordination among these agents, a distributed communication framework is often employed. However, information sharing among all agents proves to be resource-intensive, while the adoption of a manually pre-defined communication architecture imposes limitations on inter-agent communication, thereby constraining the potential for collaborative efforts. In this study, we introduce a novel approach wherein we conceptualize the communication architecture among agents as a learnable graph. We formulate this problem as the task of determining the communication graph while enabling the architecture parameters to update normally, thus necessitating a bi-level optimization process. Utilizing continuous relaxation of the graph representation and incorporating attention units, our proposed approach, CommFormer, efficiently optimizes the communication graph and concurrently refines architectural parameters through gradient descent in an end-to-end manner. Extensive experiments on a variety of cooperative tasks substantiate the robustness of our model across diverse cooperative scenarios, where agents are able to develop more coordinated and sophisticated strategies regardless of changes in the number of agents.

Motivation & Objective

• Address the limitations of manually pre-defined communication architectures in multi-agent reinforcement learning (MARL), which often lead to suboptimal performance due to high variance and lack of adaptability.
• Overcome the inefficiency and high cost of full communication among all agents in large-scale MARL, especially under bandwidth and computational constraints.
• Enable dynamic, adaptive communication structures that evolve during training to support more coordinated and scalable agent collaboration.
• Formulate the communication architecture learning as a bi-level optimization problem, allowing joint optimization of the graph structure and model parameters.
• Achieve sample-efficient, end-to-end training of communication graphs while maintaining computational feasibility and scalability across varying numbers of agents.

Proposed method

• Model the communication architecture among agents as a differentiable, learnable graph using continuous relaxation of discrete graph representations.
• Formulate the problem as a bi-level optimization: the upper-level optimizes the communication graph structure, while the lower-level trains the agent policies with fixed graph architecture.
• Integrate learnable attention mechanisms within the graph modeling framework to dynamically weight message importance and improve coordination.
• Control communication sparsity via a hyperparameter 𝒮 that limits the number of edges to 𝒮×N², ensuring bandwidth efficiency and scalability.
• Use gradient descent to jointly optimize both the communication graph and policy parameters in an end-to-end manner, avoiding discrete search over candidate architectures.
• Apply continuous relaxation to transform the discrete graph search into a continuous optimization problem, enabling backpropagation through the graph structure.

Experimental results

Research questions

• RQ1 Can a learnable graph-based communication architecture outperform manually pre-defined communication topologies in multi-agent cooperative tasks?
• RQ2 How does the sparsity of the communication graph affect performance across different MARL environments and agent counts?
• RQ3 To what extent can end-to-end optimization of communication structure and policy parameters improve coordination and scalability in MARL?
• RQ4 Does continuous relaxation of graph representation enable effective and efficient learning of optimal communication patterns without discrete search?
• RQ5 Can the proposed method generalize across diverse cooperative tasks and maintain high performance regardless of the number of agents?

Key findings

• CommFormer consistently outperforms manually pre-defined communication architectures across multiple StarCraft II micromanagement (SMAC) environments, as shown in Figure 4, demonstrating its ability to autonomously discover optimal communication patterns.
• Performance improves with increasing communication sparsity (𝒮) in complex environments, particularly as the number of agents grows, indicating that higher connectivity is essential for scalability in challenging tasks.
• In simpler tasks like 1c3s5z, even low sparsity (e.g., one communication link per agent) suffices to achieve 100% win rate, highlighting the method’s adaptability to task complexity.
• CommFormer approaches the performance upper bound of full communication (all-to-all) in scenarios where unrestricted information sharing is allowed, indicating near-optimal coordination.
• Ablation studies confirm that random pre-defined architectures lead to significant performance variance, underscoring the necessity of automated architecture search.
• The continuous relaxation of graph representation enables effective, sample-efficient, and end-to-end training, avoiding the computational burden of discrete architecture search.