[Paper Review] Learning to Gather without Communication

El Mahdi El Mhamdi, Rachid Guerraoui+2 more|arXiv (Cornell University)|Feb 21, 2018

Reinforcement Learning in Robotics39 references3 citations

TL;DR

This paper presents the first experimental evidence that multi-agent systems can learn to gather without communication using decentralized reinforcement learning. Agents on a 1D ring learn optimal gathering behavior via Q-learning with state approximation, achieving fast, robust, and scalable convergence—even tolerating up to 90% agent loss—while matching or outperforming deterministic algorithms under identical perception constraints.

ABSTRACT

A standard belief on emerging collective behavior is that it emerges from simple individual rules. Most of the mathematical research on such collective behavior starts from imperative individual rules, like always go to the center. But how could an (optimal) individual rule emerge during a short period within the group lifetime, especially if communication is not available. We argue that such rules can actually emerge in a group in a short span of time via collective (multi-agent) reinforcement learning, i.e learning via rewards and punishments. We consider the gathering problem: several agents (social animals, swarming robots...) must gather around a same position, which is not determined in advance. They must do so without communication on their planned decision, just by looking at the position of other agents. We present the first experimental evidence that a gathering behavior can be learned without communication in a partially observable environment. The learned behavior has the same properties as a self-stabilizing distributed algorithm, as processes can gather from any initial state (and thus tolerate any transient failure). Besides, we show that it is possible to tolerate the brutal loss of up to 90\% of agents without significant impact on the behavior.

Motivation & Objective

• To investigate whether agents can learn to gather in a decentralized, communication-free environment using reinforcement learning.
• To design a scalable learning framework that remains effective despite combinatorial state explosion from multiple agents.
• To evaluate the robustness of learned behavior under high agent loss (up to 90%).
• To compare the performance of learned behavior with a constrained deterministic algorithm (Q-deterministic).
• To demonstrate transferability of learned policies across different group sizes (e.g., from 10 to 100 agents) without retraining.

Proposed method

• Agents use Q-learning with eligibility traces to learn optimal actions based on rewards for group cohesion and penalties for isolation.
• A state approximation technique groups nearby agents into clusters, reducing the effective state space and enabling scalability independent of agent count.
• Each agent perceives only relative positions of others within discrete sectors, simulating limited sensory perception.
• The action space is discretized, with agents choosing movement actions (e.g., left, right, stay) based on learned Q-values.
• The learning process is fully decentralized: no central coordination or communication occurs between agents.
• A Q-deterministic baseline is created by computing Q-values along trajectories of a known deterministic barycenter algorithm under the same perception constraints for fair comparison.

Experimental results

Research questions

• RQ1 Can agents learn to gather without communication using only local rewards and punishments?
• RQ2 Can a learned policy maintain performance and stability when scaled to significantly larger groups (e.g., 100 agents) without retraining?
• RQ3 How does the performance of the learned behavior compare to a deterministic algorithm under identical perception and action constraints?
• RQ4 To what extent can the learned behavior tolerate catastrophic failures such as the loss of up to 90% of agents?
• RQ5 Can perception approximations (e.g., clustering) preserve learning efficiency and accuracy in high-dimensional or large-scale multi-agent systems?

Key findings

• Agents learned to gather in an average of 5 steps from random initial positions on a 13-node ring, with high cohesion maintained afterward.
• The learned policy generalized directly to 100 agents without retraining, achieving an average gathering time of 10 steps and a worst-case time of less than 50 steps.
• The learned behavior outperformed the Q-deterministic baseline in terms of neighbor density: average maximum neighbors reached 87% (vs. 80%) and average minimum neighbors reached 40% (vs. 28%).
• The system demonstrated strong fault tolerance, maintaining effective gathering even after the loss of up to 90% of agents.
• The standard deviation in gathering time was slightly higher for the learned behavior (5.1) than for the Q-deterministic algorithm (4.9), but the overall performance was comparable or better in key metrics.
• The use of cluster-based perception approximation enabled a constant learning space size, making the method scalable and independent of the number of agents and ring size.