[Paper Review] The Replica Dataset: A Digital Replica of Indoor Spaces

Julian Straub, Thomas J. Whelan+28 more|arXiv (Cornell University)|Jun 13, 2019

Robotics and Sensor-Based Localization27 references382 citations

TL;DR

Replica provides 18 highly photorealistic 3D indoor scene reconstructions with HDR textures, semantic/instance annotations, and reflectors, designed for ML and embodied AI research; it is Habitat-compatible and comes with a minimal SDK.

ABSTRACT

We introduce Replica, a dataset of 18 highly photo-realistic 3D indoor scene reconstructions at room and building scale. Each scene consists of a dense mesh, high-resolution high-dynamic-range (HDR) textures, per-primitive semantic class and instance information, and planar mirror and glass reflectors. The goal of Replica is to enable machine learning (ML) research that relies on visually, geometrically, and semantically realistic generative models of the world - for instance, egocentric computer vision, semantic segmentation in 2D and 3D, geometric inference, and the development of embodied agents (virtual robots) performing navigation, instruction following, and question answering. Due to the high level of realism of the renderings from Replica, there is hope that ML systems trained on Replica may transfer directly to real world image and video data. Together with the data, we are releasing a minimal C++ SDK as a starting point for working with the Replica dataset. In addition, Replica is `Habitat-compatible', i.e. can be natively used with AI Habitat for training and testing embodied agents.

Motivation & Objective

• Create a high-fidelity, photo-realistic 3D indoor dataset with rich geometry, textures, and semantics.
• Include per-primitive semantic and instance annotations and reflective surfaces (glass/mirrors).
• Enable ML research in embodied AI, navigation, and perception with realistic rendering and real-world transfer potential.
• Provide tooling (SDK) and Habitat compatibility to facilitate ML workflow and integration.

Proposed method

• Capture 3D indoor scenes with a custom RGB-D rig and IR projector for SLAM-based 6-DoF poses.
• Fuse depth into TSDF and extract meshes with Marching Cubes and texture with HDR PTex-like textures.
• Manually fix holes and planar reflectors to improve mesh quality and render realism.
• Perform two-stage semantic annotation: 2D image-based labeling on rendered views followed by fusion into a 3D segmentation forest.
• Annotate anonymizable regions and provide a hierarchical segmentation structure enabling class and instance labeling.
• Publish a minimal C++ SDK and expose data in Habitat-compatible format for easy ML integration.

Experimental results

Research questions

• RQ1 Can highly realistic 3D indoor reconstructions with HDR textures and reflectors reduce the sim-to-real domain gap for ML models?
• RQ2 How does the inclusion of planar mirror and glass reflectors impact photorealism and semantic accuracy in renderings?
• RQ3 What is the quality and structure of semantic and instance annotations, and how can they be leveraged by 3D/2D tasks and embodied AI benchmarks?
• RQ4 How does Replica compare to existing reconstruction-based datasets (e.g., Matterport3D, ScanNet) in terms of realism and annotation detail?
• RQ5 How can Replica be used with Habitat to train and test embodied agents for navigation, instruction following, and question answering?

Key findings

• Replica comprises 18 scenes with dense geometry, HDR textures, semantic class/instance annotations, and reflectors (glass/mirrors).
• HDR textures yield an exceptionally high dynamic range (~85,000:1, >16 f-stops) compared to prior datasets.
• The dataset introduces renderable reflectors and a high level of semantic precision at object boundaries, enabling accurate instance and class segmentation.
• Replica demonstrates higher color/geometry/semantic resolution relative to some prior reconstruction-based datasets (e.g., MP3D, ScanNet) as discussed in the text.
• A minimal C++ SDK and Habitat compatibility are provided to facilitate rendering, experimentation, and ML workflows.
• Six FRL apartment scenes are presented with different configurations to capture time-varying arrangements of the same space.