[논문 리뷰] ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data

Gilad Baruch, Zhuoyuan Chen외 9명|arXiv (Cornell University)|2021. 11. 16.

3D Surveying and Cultural Heritage인용 수 33

한 줄 요약

ARKitScenes는 Apple의 LiDAR가 핸드헬드 기기에서 캡처한 최초의 대규모 실내 RGB-D 데이터셋으로, 5,048 시퀀스와 1,661개 장면, 고해상도 그라운드 트루스 깊이 및 3D 바운딩 박스를 특징으로 하며, 3D 객체 감지 및 컬러 가이드 깊이 업샘플링 벤치마크를 제공합니다.

ABSTRACT

Scene understanding is an active research area. Commercial depth sensors, such as Kinect, have enabled the release of several RGB-D datasets over the past few years which spawned novel methods in 3D scene understanding. More recently with the launch of the LiDAR sensor in Apple's iPads and iPhones, high quality RGB-D data is accessible to millions of people on a device they commonly use. This opens a whole new era in scene understanding for the Computer Vision community as well as app developers. The fundamental research in scene understanding together with the advances in machine learning can now impact people's everyday experiences. However, transforming these scene understanding methods to real-world experiences requires additional innovation and development. In this paper we introduce ARKitScenes. It is not only the first RGB-D dataset that is captured with a now widely available depth sensor, but to our best knowledge, it also is the largest indoor scene understanding data released. In addition to the raw and processed data from the mobile device, ARKitScenes includes high resolution depth maps captured using a stationary laser scanner, as well as manually labeled 3D oriented bounding boxes for a large taxonomy of furniture. We further analyze the usefulness of the data for two downstream tasks: 3D object detection and color-guided depth upsampling. We demonstrate that our dataset can help push the boundaries of existing state-of-the-art methods and it introduces new challenges that better represent real-world scenarios.

연구 동기 및 목표

• Introduce ARKitScenes, a large-scale, diverse indoor RGB-D dataset captured with Apple’s LiDAR on iPad Pro/iPhone.
• Provide ground-truth depth maps registered to mobile RGB-D frames via high-precision laser scans.
• Annotate oriented 3D bounding boxes for 17 room-defining furniture categories.
• Evaluate ARKitScenes on downstream tasks: 3D object detection and color-guided depth upsampling to benchmark generalization to real-world scenarios.

제안 방법

• Capture data with iPad Pro leveraging ARKit for RGB and LiDAR-based depth plus IMU; acquire high-resolution laser scans (Faro Focus S70) for ground truth.
• Spatially register laser scans into a common venue coordinate system using Faro Scene.
• Register handheld camera frames to laser-scanned geometry by rendering synthetic views and performing 2D-3D feature matching with RANSAC and PnP, followed by dense photometric refinement.
• Manually annotate 3D oriented bounding boxes for 17 furniture categories on ARKit reconstructions.
• Provide 80/10/10 train/validation/test split based on venue-level partition to ensure non-overlapping scenes across splits.

실험 결과

연구 질문

• RQ1 How does ARKitScenes bridge the domain gap between consumer mobile depth sensing and high-quality ground-truth depth for indoor scenes?
• RQ2 Can state-of-the-art 3D object detection and depth upsampling methods trained on ARKitScenes generalize to real-world indoor environments captured with mobile LiDAR?
• RQ3 What challenges do real-world ARKitScenes pose to existing indoor 3D scene understanding methods in terms of diversity, occlusions, and depth artifacts?

주요 결과

• ARKitScenes is the largest indoor RGB-D dataset collected with a mobile device using Apple’s LiDAR, comprising 5,048 sequences across 1,661 scenes.
• The dataset includes high-resolution ground-truth depth maps registered to mobile RGB-D frames obtained via precise joint registration with Faro laser scans.
• For 3D object detection, whole-scene evaluation shows competitive performance with VoteNet, H3DNet, and MLCVNet, illustrating the dataset’s utility for indoor object localization across 17 furniture categories.
• For color-guided depth upsampling, modern DNN-based methods (MSG and MSPF) trained on ARKitScenes achieve sharper edges and better structure than classical approaches, highlighting the dataset’s realism and usefulness for depth enhancement tasks.
• The authors provide a per-venue train/validation/test split to enable standardized benchmarking on ARKitScenes.