[Paper Review] ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes

Angela Dai, Anne Lynn S. Chang+4 more|arXiv (Cornell University)|Feb 14, 2017

Robotics and Sensor-Based Localization27 references526 citations

TL;DR

ScanNet introduces a large-scale RGB-D dataset of 1513 scans with dense 3D reconstructions, camera poses, and instance-level semantic annotations, enabling supervised learning for 3D scene understanding tasks and new benchmarks.

ABSTRACT

A key requirement for leveraging supervised deep learning methods is the availability of large, labeled datasets. Unfortunately, in the context of RGB-D scene understanding, very little data is available -- current datasets cover a small range of scene views and have limited semantic annotations. To address this issue, we introduce ScanNet, an RGB-D video dataset containing 2.5M views in 1513 scenes annotated with 3D camera poses, surface reconstructions, and semantic segmentations. To collect this data, we designed an easy-to-use and scalable RGB-D capture system that includes automated surface reconstruction and crowdsourced semantic annotation. We show that using this data helps achieve state-of-the-art performance on several 3D scene understanding tasks, including 3D object classification, semantic voxel labeling, and CAD model retrieval. The dataset is freely available at http://www.scan-net.org.

Motivation & Objective

• Demonstrate that Crowd-sourced, dense RGB-D data can scale up 3D scene understanding research.
• Provide a workflow and open framework for easy capture, automatic reconstruction, and semantic annotation by non-experts.
• Show that ScanNet enables state-of-the-art performance on 3D object classification, semantic voxel labeling, and CAD model retrieval.
• Offer large-scale benchmarks and open-source tools to the research community.

Proposed method

• Design a scalable RGB-D capture system using commodity hardware (Structure sensor on iPad) with calibration using a checkerboard pattern.
• Use BundleFusion-based dense reconstruction to obtain camera poses and TSDF-based high-resolution meshes.
• Automatically align reconstructions to a common coordinate frame and extract oriented, clean meshes.
• Crowdsource instance-level semantic labeling via a WebGL annotation interface and 3D CAD model alignment via an assisted retrieval/placement interface.
• Create three benchmark tasks (3D object classification, semantic voxel labeling, CAD model retrieval) and provide train/test splits and evaluation metrics.
• Provide an open-source acquisition and annotation framework for dense RGB-D reconstructions.

Experimental results

Research questions

• RQ1 Can a novice-friendly RGB-D capture pipeline support scalable, richly annotated 3D reconstructions of real-world indoor scenes?
• RQ2 Does leveraging the ScanNet data improve deep learning-based 3D scene understanding tasks across object classification, voxel labeling, and CAD model retrieval?

Key findings

• The ScanNet dataset contains 1513 RGB-D scans from 707 distinct spaces, with 2.5M RGB-D frames, camera poses, surface reconstructions, textured meshes, and dense instance-level semantic labels.
• A crowdsourcing workflow annotates surfaces with instance-level categories and aligns CAD models to reconstructions, enabling scalable 3D annotations (681 CAD model instances across 107 annotations on 52 scans).
• Semantic voxel labeling on ScanNet reaches 73.0% voxel-class accuracy on test scenes using geometry alone (no color).
• 3D object classification benchmarks show improved transfer when training on ScanNet data (especially when combined with ShapeNet) versus synthetic data alone; mixing real ScanNet with ShapeNet improves performance.
• 3D model retrieval shows that joint training on ShapeNet and ScanNet produces strong embedding performance for real-to-synthetic model retrieval.