[Paper Review] Unity: A General Platform for Intelligent Agents

Arthur Juliani, Vincent-Pierre Berges+9 more|arXiv (Cornell University)|Sep 7, 2018

Reinforcement Learning in Robotics78 references558 citations

TL;DR

The paper argues that modern game engines, exemplified by Unity and the Unity ML-Agents Toolkit, can serve as general platforms for creating rich, configurable AI learning environments, and it surveys how this enables diverse RL research.

ABSTRACT

Recent advances in artificial intelligence have been driven by the presence of increasingly realistic and complex simulated environments. However, many of the existing environments provide either unrealistic visuals, inaccurate physics, low task complexity, restricted agent perspective, or a limited capacity for interaction among artificial agents. Furthermore, many platforms lack the ability to flexibly configure the simulation, making the simulated environment a black-box from the perspective of the learning system. In this work, we propose a novel taxonomy of existing simulation platforms and discuss the highest level class of general platforms which enable the development of learning environments that are rich in visual, physical, task, and social complexity. We argue that modern game engines are uniquely suited to act as general platforms and as a case study examine the Unity engine and open source Unity ML-Agents Toolkit. We then survey the research enabled by Unity and the Unity ML-Agents Toolkit, discussing the kinds of research a flexible, interactive and easily configurable general platform can facilitate.

Motivation & Objective

• Propose a taxonomy of simulation platforms by potential for environmental complexity (sensory, physical, task logic, social).
• Assess Unity and the Unity ML-Agents Toolkit as a general platform enabling rich, configurable AI research environments.
• Survey existing research enabled by Unity/ML-Agents and identify progress bottlenecks and opportunities for general-platform-driven advances.

Proposed method

• Introduce a four-axis taxonomy of simulators: sensory, physical, task logic, social complexity.
• Analyze Unity engine properties and how they realize environment axes (graphics, physics, scripting, multi-agent support).
• Present the Unity ML-Agents Toolkit architecture (Agents, Academy, Sensors) and the Python API for environment interaction.
• Describe the ML-Agents SDK and its components (policies, behaviors, rewards) and how they integrate with Unity scenes.
• Provide performance benchmarks and discuss curriculum learning, domain randomization, and extensibility (ICM, LSTM) within Unity environments.

Experimental results

Research questions

• RQ1 How can game engines serve as general platforms for AI research with rich sensory, physical, task, and social complexity?
• RQ2 What capabilities does Unity offer to create flexible, configurable learning environments for RL?
• RQ3 What is the architecture and workflow of Unity ML-Agents for deploying and training agents in Unity environments?
• RQ4 What research potentials and limitations exist when using a general platform like Unity for RL benchmarks and experiments?

Key findings

• Unity enables high-fidelity visuals and flexible physical simulation via PhysX/Havok (and optional third-party engines).
• The ML-Agents Toolkit provides a reusable SDK with Agents, Academy, sensors, and a Python API for training with RL and imitation learning methods.
• Simulations can run faster-than-real-time and be distributed, with options to render or skip rendering to boost speed.
• Curriculum learning and domain randomization are supported via runtime environment parameter changes and re-sampling in the Academy.
• A gym-compatible Python interface facilitates integration with existing RL workflows and benchmarks.
• The platform supports multi-agent collaboration and competition, self-play, and augmentation with modules like ICM and LSTM for richer learning signals.