[Paper Review] Scalable Multi-Agent Reinforcement Learning through Intelligent Information Aggregation

Siddharth Nayak, Kenneth Choi+4 more|arXiv (Cornell University)|Nov 3, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes InforMARL, a graph neural network-based architecture that enables scalable, decentralized multi-agent reinforcement learning by intelligently aggregating local neighborhood information. It achieves superior sample efficiency and generalization across varying numbers of agents—outperforming global-information baselines—while maintaining high success rates in navigation and coordination tasks with only local observations.

ABSTRACT

We consider the problem of multi-agent navigation and collision avoidance when observations are limited to the local neighborhood of each agent. We propose InforMARL, a novel architecture for multi-agent reinforcement learning (MARL) which uses local information intelligently to compute paths for all the agents in a decentralized manner. Specifically, InforMARL aggregates information about the local neighborhood of agents for both the actor and the critic using a graph neural network and can be used in conjunction with any standard MARL algorithm. We show that (1) in training, InforMARL has better sample efficiency and performance than baseline approaches, despite using less information, and (2) in testing, it scales well to environments with arbitrary numbers of agents and obstacles. We illustrate these results using four task environments, including one with predetermined goals for each agent, and one in which the agents collectively try to cover all goals. Code available at https://github.com/nsidn98/InforMARL.

Motivation & Objective

• To address the challenge of scalable multi-agent navigation with limited local observations and decentralized execution.
• To improve sample efficiency and policy transferability in multi-agent reinforcement learning without relying on global state information.
• To demonstrate that intelligent aggregation of local information can outperform naive global state concatenation in MARL.
• To enable effective coordination and collision avoidance in dense, dynamic environments using only local observations.
• To show that graph-based information aggregation enhances scalability and performance in diverse multi-agent navigation tasks.

Proposed method

• Incorporates a graph neural network (GNN) to encode and aggregate local neighborhood information for both actor and critic networks in a MARL framework.
• Represents the environment as a graph where agents and obstacles are nodes, and edges encode proximity-based relationships.
• Uses GNN message-passing to compute a context-aware, aggregated representation of local observations for each agent.
• Integrates the GNN-processed information into standard MARL algorithms (e.g., RMAPPO) without modifying the core training paradigm.
• Enables decentralized execution by ensuring each agent acts based only on its local observation and the aggregated neighborhood vector.
• Supports transferability by training on a fixed number of agents and testing on different numbers without retraining.

Experimental results

Research questions

• RQ1 Can intelligent aggregation of local information improve sample efficiency in multi-agent reinforcement learning compared to global state baselines?
• RQ2 Does a GNN-based information aggregation module enable better generalization to unseen numbers of agents than standard MARL algorithms?
• RQ3 Can a decentralized MARL policy trained with only local observations achieve high performance in complex coordination tasks like coverage and formation?
• RQ4 Does using less information (local only) lead to better scalability and lower sample complexity than using global state?
• RQ5 Can graph-based representations effectively identify and utilize the most relevant environmental information for navigation and collision avoidance?

Key findings

• InforMARL achieved 100% success rate across all tested environments (Target, Coverage, Formation, Line) with both 3 and 7 agents, even when trained only on 3-agent scenarios.
• In the Target environment, InforMARL reached goals in approximately 39% of the episode length (T ≈ 0.39), demonstrating efficient navigation.
• In the Coverage and Formation tasks, InforMARL completed tasks in 0.30–0.43 of the episode length, matching or slightly outperforming RMAPPO despite using only local observations.
• InforMARL showed better sample efficiency during training than baseline methods, even though it used less information than global-state models.
• The method generalized effectively to different numbers of agents, with no performance drop when testing on 7 agents after training on 3.
• InforMARL maintained high performance across all tasks, including those requiring consensus on goals (e.g., Formation, Line), proving its ability to support complex coordination with minimal information.