[Paper Review] Emergent collective intelligence from massive-agent cooperation and competition

Hanmo Chen, Stone Tao+7 more|arXiv (Cornell University)|Jan 4, 2023

Evolutionary Game Theory and Cooperation4 citations

TL;DR

This paper introduces Lux, a massive-agent reinforcement learning environment simulating large-scale cooperation and competition among thousands of agents for limited resources and survival against darkness. Using a centralized pixel-to-pixel policy network with PPO, the authors demonstrate emergent collective intelligence—such as coordinated territory expansion and strategic movement—without explicit coordination, achieving a 90% win rate against state-of-the-art baselines and generalizing effectively to larger maps.

ABSTRACT

Inspired by organisms evolving through cooperation and competition between different populations on Earth, we study the emergence of artificial collective intelligence through massive-agent reinforcement learning. To this end, We propose a new massive-agent reinforcement learning environment, Lux, where dynamic and massive agents in two teams scramble for limited resources and fight off the darkness. In Lux, we build our agents through the standard reinforcement learning algorithm in curriculum learning phases and leverage centralized control via a pixel-to-pixel policy network. As agents co-evolve through self-play, we observe several stages of intelligence, from the acquisition of atomic skills to the development of group strategies. Since these learned group strategies arise from individual decisions without an explicit coordination mechanism, we claim that artificial collective intelligence emerges from massive-agent cooperation and competition. We further analyze the emergence of various learned strategies through metrics and ablation studies, aiming to provide insights for reinforcement learning implementations in massive-agent environments.

Motivation & Objective

• To study the emergence of artificial collective intelligence through large-scale agent cooperation and competition.
• To develop a scalable, open-source environment (Lux) supporting thousands of agents in dynamic, resource-scarce settings.
• To investigate whether complex group strategies can emerge organically from individual agent decisions without explicit coordination mechanisms.
• To evaluate the effectiveness of centralized control in massive-agent MARL by comparing it with decentralized approaches.

Proposed method

• Proposed Lux, a grid-world environment supporting up to thousands of agents per team competing for resources and fighting darkness.
• Employed a centralized control framework using a pixel-to-pixel policy network to map raw observations to actions, bypassing credit assignment issues.
• Trained agents using Proximal Policy Optimization (PPO) with curriculum learning phases to stabilize training.
• Designed a hierarchical feature extraction pipeline: map features via three convolutional layers, unit and city features via max-pooling and fully-connected layers.
• Implemented a hybrid observation encoding combining global, self, unit, and city features for comprehensive state representation.
• Conducted ablation studies comparing centralized vs. decentralized policies, and evaluated generalization across map sizes (12×12 to 128×128).

Experimental results

Research questions

• RQ1 Can collective intelligence emerge from massive-agent cooperation and competition without explicit coordination mechanisms?
• RQ2 How does centralized control via pixel-to-pixel policy networks improve performance and strategy emergence in large-scale multi-agent environments?
• RQ3 To what extent can policies trained on smaller maps generalize to larger maps with thousands of agents?
• RQ4 What role does curriculum learning play in enabling the acquisition of complex group strategies in massive-agent settings?
• RQ5 How do credit assignment and policy architecture affect the scalability and performance of multi-agent reinforcement learning?

Key findings

• The centralized pixel-to-pixel policy achieved a 90% win rate against the state-of-the-art policy from the Toad Brigade team in the Lux AI competition.
• Agents spontaneously developed coordinated strategies such as diagonal formation movement and territory segmentation without prior knowledge or explicit coordination.
• The centralized control approach achieved a 98% win rate over 100 episodes when compared to a decentralized policy, demonstrating superior coordination and adaptability.
• Transferred policies trained on 32×32 maps generalized effectively to 48×48 and 64×64 maps, maintaining high levels of agent coordination with over 1,000 units and CityTiles.
• On 128×128 maps, despite simulation slowdown and cooldown limitations, agents retained core strategies learned on smaller maps, indicating potential for million-agent scalability.
• Ablation studies confirmed that centralized control significantly outperforms decentralized learning in complex, large-scale scenarios due to better credit assignment and strategic coherence.