[Paper Review] Heterogeneous Graph Attention Networks for Learning Diverse Communication

Esmaeil Seraj, Zheyuan Wang+4 more|arXiv (Cornell University)|Aug 21, 2021

Advanced Graph Neural Networks43 references4 citations

TL;DR

This paper proposes HetNet, a heterogeneous graph attention network that learns diverse, efficient communication protocols for multi-agent teams with different capabilities, using a novel Multi-Agent Heterogeneous Actor-Critic (MAHAC) framework. It achieves ~10% performance improvement over baselines in heterogeneous environments by modeling inter-class communication through a shared language via GNN-based message translation.

ABSTRACT

Multi-agent teaming achieves better performance when there is communication among participating agents allowing them to coordinate their actions for maximizing shared utility. However, when collaborating a team of agents with different action and observation spaces, information sharing is not straightforward and requires customized communication protocols, depending on sender and receiver types. Without properly modeling such heterogeneity in agents, communication becomes less helpful and could even deteriorate the multi-agent cooperation performance. We propose heterogeneous graph attention networks, called HetNet, to learn efficient and diverse communication models for coordinating heterogeneous agents towards accomplishing tasks that are of collaborative nature. We propose a Multi-Agent Heterogeneous Actor-Critic (MAHAC) learning paradigm to obtain collaborative per-class policies and effective communication protocols for composite robot teams. Our proposed framework is evaluated against multiple baselines in a complex environment in which agents of different types must communicate and cooperate to satisfy the objectives. Experimental results show that HetNet outperforms the baselines in learning sophisticated multi-agent communication protocols by achieving $\sim$10\% improvements in performance metrics.

Motivation & Objective

• To address the challenge of inefficient communication in heterogeneous multi-agent teams where agents have different action, observation, and sensor capabilities.
• To develop a scalable, end-to-end communication learning framework that accounts for agent heterogeneity in both state and action spaces.
• To design a communication protocol that enables agents of different types to exchange meaningful, task-relevant information despite differing 'languages'.
• To evaluate the framework in a new Multi-Agent Heterogeneous POMDP (MAH-POMDP) environment that captures real-world team composition diversity.
• To demonstrate that modeling heterogeneity in communication leads to improved coordination and performance over standard MARL baselines.

Proposed method

• Proposes a new problem formulation: Multi-Agent Heterogeneous Partially Observable Markov Decision Process (MAH-POMDP) to model collaborative tasks with heterogeneous agents.
• Introduces HetNet, a heterogeneous graph attention network that learns inter-class message representations by encoding agent types and roles into message embeddings.
• Employs a centralized training with decentralized execution (CTDE) paradigm to train policies and communication protocols end-to-end.
• Uses a per-class critic structure in the MAHAC algorithm to enable class-wise policy learning and improve credit assignment in heterogeneous teams.
• Designs a limited-length, real-valued communication channel using GNNs to translate messages across agent classes into a shared representation.
• Applies a message-passing mechanism where each agent attends to relevant messages from other classes, enabling stylized, role-aware communication.

Experimental results

Research questions

• RQ1 Can a communication protocol be learned that effectively coordinates heterogeneous agents with different action and observation spaces?
• RQ2 How does modeling agent heterogeneity in communication improve multi-agent coordination performance compared to homogeneous or non-heterogeneous baselines?
• RQ3 What is the impact of different critic architectures (centralized, per-class, per-agent) on learning efficiency and performance in heterogeneous MARL?
• RQ4 Can the proposed framework generalize to both homogeneous and heterogeneous environments while maintaining superior performance?
• RQ5 How does the use of graph attention mechanisms enhance message relevance and diversity in inter-agent communication?

Key findings

• HetNet achieves approximately 10% improvement in performance metrics over state-of-the-art baselines like CommNet and IC3Net in both homogeneous and heterogeneous environments.
• The per-class critic structure in MAHAC leads to better learning efficiency and performance than fully centralized critics, reducing episode completion steps from 10.01 to 9.81 on average.
• HetNet with per-class and per-agent critics shows similar performance, both outperforming the centralized critic variant, indicating that class-wise credit assignment enhances learning.
• The ablation study confirms that modeling heterogeneity in communication is critical—without it, communication can degrade performance.
• The framework generalizes well, demonstrating effectiveness in both adopted homogeneous (Predator-Prey) and novel heterogeneous (Predator-Capture-Prey) environments.
• The use of GNN-based message translation enables agents to 'speak' a shared language, resolving the language barrier between heterogeneous agents.