[Paper Review] Joint 2D-3D-Semantic Data for Indoor Scene Understanding

Iro Armeni, Sasha Sax+2 more|arXiv (Cornell University)|Feb 3, 2017

3D Surveying and Cultural HeritageEarth and Planetary Sciences8 references684 citations

TL;DR

This paper introduces a large, richly annotated indoor dataset that provides mutually registered 2D (RGB, depth, normals, 360° images) and 3D (meshes, point clouds) modalities with instance-level semantic annotations across 13 object classes. It enables cross-modal and joint learning for indoor scene understanding.

ABSTRACT

We present a dataset of large-scale indoor spaces that provides a variety of mutually registered modalities from 2D, 2.5D and 3D domains, with instance-level semantic and geometric annotations. The dataset covers over 6,000m2 and contains over 70,000 RGB images, along with the corresponding depths, surface normals, semantic annotations, global XYZ images (all in forms of both regular and 360° equirectangular images) as well as camera information. It also includes registered raw and semantically annotated 3D meshes and point clouds. The dataset enables development of joint and cross-modal learning models and potentially unsupervised approaches utilizing the regularities present in large-scale indoor spaces. The dataset is available here: http://3Dsemantics.stanford.edu/

Motivation & Objective

• Motivate the need for a large-scale, multi-modality indoor dataset that supports joint and cross-modal learning.
• Provide a synchronized collection of 2D RGB, depth, normals, 360° equirectangular images, 3D meshes, and point clouds with consistent semantic annotations across modalities.
• Enable downstream tasks such as scene understanding, depth/normal estimation, object detection, segmentation, and amodal reconstruction.
• Offer data collection, processing pipelines, and baseline results to facilitate development of cross-domain and unsupervised learning approaches.

Proposed method

• Acquire six large indoor areas with Matterport scanning to obtain 3D textured meshes, point clouds, and raw RGB-D data.
• Annotate 3D point clouds with 13 object classes and 11 scene labels, then project annotations to meshes and 2D images.
• Generate 2D semantic labels by projecting 3D semantics onto the image domain and provide 3D semantic meshes.
• Produce 360° equirectangular images and sample additional regular RGB-D images through a sampling strategy guided by semantic content entropy.
• Render depth images from the 3D mesh (z-buffer) and compute surface normals from the mesh; store as 16-bit depth and 24-bit normal images.
• Provide 3D coordinate encoded OpenEXR images and a consistent naming convention; describe data collection, processing, and sampling workflow.

Experimental results

Research questions

• RQ1 How can mutually registered 2D, 2.5D, and 3D modalities be leveraged together for indoor scene understanding?
• RQ2 What is the benefit of 3D-based semantic annotations when projected to 2D and 2.5D modalities?
• RQ3 Can a large-scale indoor dataset with diverse modalities support cross-modal and unsupervised learning approaches?
• RQ4 What baseline results can be established for 3D object detection using this multi-modality data?
• RQ5 How do sampling strategies based on semantic entropy influence the diversity and usefulness of annotated images?

Key findings

• The dataset contains 70,496 regular RGB images and 1,413 equirectangular RGB images, with depths, surface normals, semantic annotations, global XYZ images, and camera metadata across six areas.
• It provides 695,878,620 3D points and 1,266,295 3D mesh faces with 13 object classes and 11 scene labels, enabling cross-modal semantic transfer and amodal analysis.
• 3D semantics are projected onto 2D semantic images, enabling ground-truth annotations in multiple domains and supporting tasks like 3D object detection and scene parsing.
• A sampling pipeline generates 72 images per scan location from equirectangular data, with camera yaw/pitch/FOV distributions and entropy-based filtering to preserve semantically diverse content.
• Baseline results are provided for 3D object detection on the 3D point clouds, including hierarchical parsing and SVM/CRF-based methods, demonstrating the dataset’s utility for evaluating cross-modal and hierarchical approaches.
• The dataset enables generation of additional data (via 3D meshes and equirectangular projections) and supports both supervised and potential unsupervised learning approaches across modalities.