[Paper Review] Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI

Santhosh Kumar Ramakrishnan, Aaron Gokaslan+11 more|arXiv (Cornell University)|Sep 16, 2021

Robotics and Sensor-Based Localization81 citations

TL;DR

HM3D is a large-scale, high-fidelity 3D indoor dataset with 1000 building-scale reconstructions, offering greater scale, completeness, and visual fidelity than prior datasets, enabling improved embodied AI navigation across multiple test environments.

ABSTRACT

We present the Habitat-Matterport 3D (HM3D) dataset. HM3D is a large-scale dataset of 1,000 building-scale 3D reconstructions from a diverse set of real-world locations. Each scene in the dataset consists of a textured 3D mesh reconstruction of interiors such as multi-floor residences, stores, and other private indoor spaces. HM3D surpasses existing datasets available for academic research in terms of physical scale, completeness of the reconstruction, and visual fidelity. HM3D contains 112.5k m^2 of navigable space, which is 1.4 - 3.7x larger than other building-scale datasets such as MP3D and Gibson. When compared to existing photorealistic 3D datasets such as Replica, MP3D, Gibson, and ScanNet, images rendered from HM3D have 20 - 85% higher visual fidelity w.r.t. counterpart images captured with real cameras, and HM3D meshes have 34 - 91% fewer artifacts due to incomplete surface reconstruction. The increased scale, fidelity, and diversity of HM3D directly impacts the performance of embodied AI agents trained using it. In fact, we find that HM3D is `pareto optimal' in the following sense -- agents trained to perform PointGoal navigation on HM3D achieve the highest performance regardless of whether they are evaluated on HM3D, Gibson, or MP3D. No similar claim can be made about training on other datasets. HM3D-trained PointNav agents achieve 100% performance on Gibson-test dataset, suggesting that it might be time to retire that episode dataset.

Motivation & Objective

• Provide a large-scale, near-complete 3D reconstruction dataset of real-world indoor buildings.
• Improve visual fidelity and reduce reconstruction artifacts relative to existing datasets.
• Demonstrate the utility of HM3D for training and evaluating embodied AI agents in navigation tasks.
• Show Pareto-optimal transfer performance of HM3D-trained agents across multiple test environments.

Proposed method

• Capture and reconstruct 1000 building-scale interiors using Matterport Pro2 scans and the Matterport pipeline.
• Quantify scale, completeness, and visual fidelity relative to prior datasets through quantitative metrics.
• Evaluate embodied AI navigation (PointGoal/PointNav) performance when trained on HM3D versus other datasets.
• Provide Habitat-ready metadata and integration to facilitate training in the Habitat simulator.

Experimental results

Research questions

• RQ1 Does HM3D offer scalable benefits for training embodied AI agents compared to existing indoor datasets?
• RQ2 How do HM3D’s scale, completeness, and fidelity impact agent navigation performance and generalization across datasets?
• RQ3 Is training on HM3D Pareto-optimal in terms of performance across Gibson, MP3D, and HM3D test sets?
• RQ4 How does visual fidelity and reconstruction completeness of HM3D compare to real-world imagery and prior datasets?
• RQ5 Can increasing navigable area in training scenes correlate with improvements in PointNav performance?

Key findings

• HM3D provides 1000 building-scale reconstructions with 112.5k m^2 navigable space, outperforming MP3D and Gibson in scale.
• HM3D shows 34.91% fewer reconstruction artifacts and 20.85% higher visual fidelity versus prior photorealistic datasets.
• HM3D-trained PointNav agents achieve state-of-the-art or Pareto-optimal performance across Gibson, MP3D, and HM3D test sets, including 100% success on Gibson test with depth inputs.
• Rendered HM3D images have significantly lower FID/KID scores compared to MP3D and Gibson real-image baselines, indicating higher visual fidelity to real imagery.
• Navigation performance scales near-linearly with the total navigable area in training HM3D scenes (rho ≈ 0.88).
• HM3D agents generalize better to harder episodes, demonstrating diverse layouts and appearance aiding transferability.