[Paper Review] AI2-THOR: An Interactive 3D Environment for Visual AI

Eric Kolve, Roozbeh Mottaghi+11 more|arXiv (Cornell University)|Dec 14, 2017

Multimodal Machine Learning Applications26 references327 citations

TL;DR

AI2-THOR is a large-scale, near photo-realistic 3D indoor simulation platform enabling embodied AI research with interactive agents, multiple scene datasets, diverse actions, and rich metadata to train and evaluate vision-and-action models.

ABSTRACT

We introduce The House Of inteRactions (THOR), a framework for visual AI research, available at http://ai2thor.allenai.org. AI2-THOR consists of near photo-realistic 3D indoor scenes, where AI agents can navigate in the scenes and interact with objects to perform tasks. AI2-THOR enables research in many different domains including but not limited to deep reinforcement learning, imitation learning, learning by interaction, planning, visual question answering, unsupervised representation learning, object detection and segmentation, and learning models of cognition. The goal of AI2-THOR is to facilitate building visually intelligent models and push the research forward in this domain.

Motivation & Objective

• Motivate visual AI research to move beyond static images by enabling interaction with a realistic 3D environment.
• Provide near photo-realistic scenes, diverse agents, and a rich action space to train and evaluate embodied AI models.
• Offer scalable, fast, and cost-effective simulation as a proxy for real-world experiments to improve generalization.

Proposed method

• Describe the AI2-THOR framework with its Unity-based 3D scenes and Python API for agent control.
• Explain scene datasets (iTHOR, RoboTHOR, ProcTHOR, ArchitecTHOR) and the role of procedural generation for generalization.
• Detail agent embodiments (ManipulaTHOR, StretchRE1, LoCoBot, Abstract, Drone) and their interaction capabilities.
• Classify actions into navigation, interaction, environment queries, and environment state changes.
• Outline image modalities (RGB, Depth, Semantic/Instance Segmentation, Normals) and object database contents (3,578 interactive objects).
• Present metadata offerings and their use in reward design, imitation learning, and evaluation datasets.

Experimental results

Research questions

• RQ1 How can a rich, interactive 3D environment accelerate learning and generalization for embodied AI compared to static datasets?
• RQ2 What combination of scenes, agents, actions, and modalities yields scalable, transferable training for visual AI tasks?
• RQ3 To what extent do procedurally generated environments (ProcTHOR) improve generalization to real-world-like scenes (ArchitecTHOR, RoboTHOR) in embodied tasks?
• RQ4 What is the performance of AI2-THOR as a simulation platform relative to other simulators in terms of scale, capabilities, and efficiency?

Key findings

• AI2-THOR supports extensive interactions (state changes, arm manipulation, causal interactions) and scales with numerous scenes and objects.
• Procedural generation (ProcTHOR-10K) enables large-scale training that improves generalization across RoboTHOR, iTHOR, and ArchitecTHOR in zero-shot settings.
• A comprehensive agent ecosystem (ManipulaTHOR, StretchRE1, LoCoBot, Abstract, Drone) supports a range of embeddings from low-level manipulation to navigation and abstracted actions.
• Rich image modalities (RGB, depth, semantic/instance segmentation, normals) and environment metadata enhance training signals and reward shaping for imitation and reinforcement learning.
• AI2-THOR has been used in over 150 publications and supports fast, scalable training with competitive performance benchmarks against other simulators (Appendix B discusses profiling and parallelization).
• A wide variety of research domains benefit from AI2-THOR, including visual navigation, audio-visual tasks, vision-and-language, sim2real, multi-agent collaboration, affordances, scene synthesis, and interpretable representations.