[Paper Review] Neural MMO: A Massively Multiagent Game Environment for Training and Evaluating Intelligent Agents

Joseph Suárez, Yilun Du+2 more|arXiv (Cornell University)|Mar 2, 2019

Reinforcement Learning in Robotics20 references48 citations

TL;DR

Neural MMO presents a persistent, procedurally generated, massively multiagent environment where neural agents learn survival via reinforcement learning, showing that larger populations promote exploration and niche formation.

ABSTRACT

The emergence of complex life on Earth is often attributed to the arms race that ensued from a huge number of organisms all competing for finite resources. We present an artificial intelligence research environment, inspired by the human game genre of MMORPGs (Massively Multiplayer Online Role-Playing Games, a.k.a. MMOs), that aims to simulate this setting in microcosm. As with MMORPGs and the real world alike, our environment is persistent and supports a large and variable number of agents. Our environment is well suited to the study of large-scale multiagent interaction: it requires that agents learn robust combat and navigation policies in the presence of large populations attempting to do the same. Baseline experiments reveal that population size magnifies and incentivizes the development of skillful behaviors and results in agents that outcompete agents trained in smaller populations. We further show that the policies of agents with unshared weights naturally diverge to fill different niches in order to avoid competition.

Motivation & Objective

• Introduce a persistent, scalable multiagent environment inspired by MMORPGs for training intelligent agents.
• Enable large populations with varying numbers of species to study emergent behaviors under competition for resources.
• Demonstrate how population size and species diversity affect exploration, specialization, and policy learning.

Proposed method

• Agents operate in tile-based, procedurally generated maps with food and water foraging and a strategic combat system.
• Policies are neural networks trained with policy gradient methods and shared or unshared weights across populations.
• Observations are local crops of the environment including tile types and agent attributes; actions include a movement and an attack choice.
• The reward signal is survival time, computed as a discounted sum of unit rewards per tick.
• Experiments use multiple world instances and server merges to evaluate performance across population settings.

Experimental results

Research questions

• RQ1 How does increasing the number of concurrent agents influence exploration and policy learning in a persistent multiagent environment?
• RQ2 How does the number of distinct populations with unshared weights affect niche formation and specialization?
• RQ3 What are the effects of environment randomization and tournament-style evaluations on learned policies under competition?
• RQ4 Do policies trained with larger populations generalize when merged and evaluated against diverse agent bases?

Key findings

• Larger population sizes consistently improve agent lifetimes in tournaments.
• Population size magnifies exploration, leading to broader map visitation.
• More populations with unshared weights promote niche formation and specialization across the map.
• Agents learn dependencies and strategies that depend on other agents’ policies and positions.
• Combat adds strong coupling among agents, further driving emergent behaviors and robust policies.