[Paper Review] Low-Bandwidth Communication Emerges Naturally in Multi-Agent Learning Systems

Niko A. Grupen, Daniel D. Lee+1 more|arXiv (Cornell University)|Nov 30, 2020

Evolutionary Game Theory and Cooperation38 references4 citations

TL;DR

This paper demonstrates that low-bandwidth communication—such as implicit coordination through movement rules—naturally emerges in multi-agent reinforcement learning (MARL) systems during cooperative pursuit-evasion tasks. Using a potential field-based predator strategy and MARL algorithms, the authors show that agents develop efficient, low-complexity coordination strategies resembling animal foraging behaviors, without explicit language or high-bandwidth signaling.

ABSTRACT

In this work, we study emergent communication through the lens of cooperative multi-agent behavior in nature. Using insights from animal communication, we propose a spectrum from low-bandwidth (e.g. pheromone trails) to high-bandwidth (e.g. compositional language) communication that is based on the cognitive, perceptual, and behavioral capabilities of social agents. Through a series of experiments with pursuit-evasion games, we identify multi-agent reinforcement learning algorithms as a computational model for the low-bandwidth end of the communication spectrum.

Motivation & Objective

• To understand how low-bandwidth communication arises in multi-agent systems through insights from animal communication.
• To investigate whether multi-agent reinforcement learning (MARL) algorithms can give rise to emergent, low-complexity communication strategies.
• To model the spectrum of communication in nature—from pheromone trails to vocal signals—within a computational reinforcement learning framework.
• To evaluate whether MARL agents develop coordination mechanisms resembling natural low-bandwidth interaction rules, such as those seen in wolf packs or dolphin herds.
• To demonstrate that high-level coordination can emerge without explicit language or symbolic representation, relying instead on sensorimotor feedback and reward maximization.

Proposed method

• The authors use a pursuit-evasion game on a toroidal grid environment with multiple predators and one prey.
• Predators use a potential field-based strategy to compute optimal pursuit directions based on prey and predator positions, using a cost function that incorporates distance and angular alignment.
• The optimization of the potential field function is performed over unrolled replications of the torus to handle periodic boundary conditions, enabling discrete search over possible predator configurations.
• The system selects the set of 'active' predators that maximize the potential field function, forming dynamic, symmetric pursuit formations.
• MARL agents are trained to coordinate using policy gradients, with the goal of capturing the prey while minimizing time-to-capture.
• The method incorporates a gradient-based update rule that adjusts predator headings based on the negative gradient of the potential field, favoring symmetric, stable formations.

Experimental results

Research questions

• RQ1 Can multi-agent reinforcement learning systems naturally develop low-bandwidth communication strategies without explicit design?
• RQ2 To what extent do MARL agents replicate known animal foraging and herding behaviors, such as those seen in wolves or dolphins?
• RQ3 How do physical and environmental constraints (e.g., limited sensing, discrete action spaces) shape the emergence of communication in cooperative agents?
• RQ4 Does the absence of symbolic or compositional language in MARL systems still lead to effective, coordinated behavior?
• RQ5 Can implicit, non-linguistic coordination mechanisms (e.g., interaction rules) emerge from reward maximization in partially observable environments?

Key findings

• Low-bandwidth communication strategies, such as symmetric pursuit formations and reactive positional adjustments, emerge naturally in MARL systems during pursuit-evasion tasks.
• Predators learn to coordinate their movements to maintain group cohesion and avoid disintegration, even when temporarily moving away from the prey to reposition.
• The potential field-based strategy successfully generates stable, symmetric formations that mimic real animal herding behaviors observed in wolves and dolphins.
• The learned behavior shows qualitative similarity to documented animal foraging strategies, including coordinated encirclement and dynamic repositioning.
• The absence of explicit language or symbolic communication does not hinder effective coordination, as agents achieve high capture rates through implicit, low-bandwidth interaction rules.
• The method scales reasonably for small numbers of agents (N ≤ 3) and small unrolling factors (k = 1), demonstrating feasibility for modeling emergent coordination.