[Paper Review] Megaverse: Simulating Embodied Agents at One Million Experiences per Second

Aleksei Petrenko, Erik Wijmans+2 more|arXiv (Cornell University)|Jul 17, 2021

Virtual Reality Applications and Impacts4 citations

TL;DR

Megaverse is a high-throughput 3D simulation platform for embodied AI and reinforcement learning, achieving over 1 million experiences per second on a single 8-GPU node by leveraging batched rendering and physics simulation. It enables fast, scalable training of single- and multi-agent agents in immersive, procedurally generated environments, with a new benchmark (Megaverse-8) that challenges advanced skills like navigation, memory, and object manipulation.

ABSTRACT

We present Megaverse, a new 3D simulation platform for reinforcement learning and embodied AI research. The efficient design of our engine enables physics-based simulation with high-dimensional egocentric observations at more than 1,000,000 actions per second on a single 8-GPU node. Megaverse is up to 70x faster than DeepMind Lab in fully-shaded 3D scenes with interactive objects. We achieve this high simulation performance by leveraging batched simulation, thereby taking full advantage of the massive parallelism of modern GPUs. We use Megaverse to build a new benchmark that consists of several single-agent and multi-agent tasks covering a variety of cognitive challenges. We evaluate model-free RL on this benchmark to provide baselines and facilitate future research. The source code is available at https://www.megaverse.info

Motivation & Objective

• To address the inefficiency of game-engine-based RL simulators that underutilize GPU parallelism and require frame-skipping.
• To accelerate deep reinforcement learning research by enabling high-throughput, low-latency simulation of embodied agents in 3D environments.
• To democratize access to large-scale, high-fidelity simulation by reducing computational costs while maintaining rich, interactive scenes.
• To support complex, generalizable tasks requiring memory, navigation, and physics-based manipulation for advancing embodied intelligence.
• To enable scalable multi-agent training with minimal performance degradation, supporting cooperative and competitive settings.

Proposed method

• The platform uses batched rendering and physics simulation to fully exploit GPU parallelism, eliminating frame-skipping and enabling high experience collection rates.
• It simulates physics at a lower frequency than traditional engines, reducing computational overhead while maintaining sufficient realism for learning.
• The engine natively supports multiple agents in the same environment without performance loss, enabling efficient multi-agent training.
• It generates procedurally diverse environments to test generalization and robustness of trained agents.
• A new benchmark, Megaverse-8, consists of eight tasks covering navigation, memory, exploration, and manipulation, all with high-dimensional egocentric observations.
• Reward shaping techniques like Team Spirit are used to encourage cooperation in multi-agent settings, with curriculum-based gradual introduction.

Experimental results

Research questions

• RQ1 Can a purpose-built simulation engine achieve over 1 million RL experiences per second on a single 8-GPU node while supporting complex 3D, interactive environments?
• RQ2 How does batched simulation improve throughput and resource utilization compared to traditional game-engine-based RL environments?
• RQ3 To what extent can fast simulation enable scalable multi-agent training in collaborative and competitive settings without performance degradation?
• RQ4 Can fast simulation accelerate the evaluation of advanced optimization methods like derivative-free optimization in high-dimensional 3D environments?
• RQ5 How effective are reward shaping strategies like Team Spirit in enabling cooperation among agents in complex, multi-agent tasks?

Key findings

• Megaverse achieves 1.1 million observations per second on a single 8-GPU node, representing a 70x speedup over DeepMind Lab in fully-shaded 3D scenes with interactive objects.
• The platform enables simulation at up to 20x faster than Atari and 70x faster than DeepMind Lab on standard research hardware, without frame-skipping.
• Multi-agent training scales efficiently: dozens of agents can be simulated simultaneously in the same environment with no performance loss.
• In tasks like HexExplore and Collect, multiple agents improve performance, but Team Spirit reward shaping hurts performance due to increased credit assignment difficulty.
• In TowerBuilding, Team Spirit is essential for cooperation; without it, agents compete and performance degrades with more agents.
• The Megaverse-8 benchmark includes eight procedurally generated tasks that require advanced skills like memory, object manipulation, and composition of low-level behaviors, forming a hard challenge for current RL algorithms.