[Paper Review] Capturing, Reconstructing, and Simulating: the UrbanScene3D Dataset

Liqiang Lin, Yilin Liu+4 more|arXiv (Cornell University)|Jul 9, 2021

Remote Sensing and LiDAR Applications4 citations

TL;DR

UrbanScene3D introduces a large-scale, multi-source urban scene dataset combining real-world aerial captures and synthetic simulations in Unreal Engine with AirSim, enabling high-fidelity 3D reconstruction, aerial path planning benchmarking, and autonomous agent simulation. The dataset includes 128k+ high-resolution images, LiDAR scans, and detailed instance-level annotations across 16 scenes totaling 136 km², supporting evaluation of path planning and MVS algorithms with ground-truth accuracy and completeness metrics.

ABSTRACT

We present UrbanScene3D, a large-scale data platform for research of urban scene perception and reconstruction. UrbanScene3D contains over 128k high-resolution images covering 16 scenes including large-scale real urban regions and synthetic cities with 136 km^2 area in total. The dataset also contains high-precision LiDAR scans and hundreds of image sets with different observation patterns, which provide a comprehensive benchmark to design and evaluate aerial path planning and 3D reconstruction algorithms. In addition, the dataset, which is built on Unreal Engine and Airsim simulator together with the manually annotated unique instance label for each building in the dataset, enables the generation of all kinds of data, e.g., 2D depth maps, 2D/3D bounding boxes, and 3D point cloud/mesh segmentations, etc. The simulator with physical engine and lighting system not only produce variety of data but also enable users to simulate cars or drones in the proposed urban environment for future research.

Motivation & Objective

• To address the lack of comprehensive, large-scale urban scene datasets that combine real-world and synthetic data for 3D perception and reconstruction research.
• To provide a benchmark for evaluating aerial path planning strategies in urban environments using high-precision LiDAR scans and MVS reconstruction results.
• To enable realistic simulation of drones and cars in diverse urban environments with controllable lighting, weather, and physical dynamics.
• To support data-driven research in 3D instance segmentation, depth estimation, and novel view synthesis through rich annotations and multi-modal data.
• To bridge the domain gap between synthetic simulations and real-world urban scenes by integrating high-fidelity real reconstructions with physically accurate simulations.

Proposed method

• The dataset is constructed using multi-view stereo (MVS) techniques on drone-captured aerial images to generate detailed, textured 3D meshes of real urban regions.
• High-precision LiDAR scans are used as ground-truth for evaluating reconstruction accuracy and completeness in the benchmark.
• A custom simulation platform is built on Unreal Engine and AirSim, enabling physics-based simulation of drones and cars with realistic lighting and weather control.
• Each building in the scene is manually annotated with a unique instance ID, enabling 3D instance segmentation and multi-view data generation.
• A variety of aerial path planning algorithms are implemented and evaluated, including oblique photography, Smith et al. [37], Zhou et al. [46], and Zhang et al. [44], to compare energy efficiency and reconstruction quality.
• The simulator generates diverse data types such as 2D depth maps, 2D/3D bounding boxes, and 3D point cloud/mesh segmentations for downstream tasks.

Experimental results

Research questions

• RQ1 How do different aerial path planning strategies impact the quality and energy efficiency of 3D urban scene reconstruction?
• RQ2 To what extent can synthetic simulations in UrbanScene3D generalize to real-world 3D reconstruction and perception tasks?
• RQ3 What is the reconstruction accuracy and completeness of MVS pipelines when evaluated against high-precision LiDAR ground-truth?
• RQ4 How effective is the UrbanScene3D simulator in generating realistic, diverse data for autonomous navigation and 3D perception tasks?
• RQ5 Can the inclusion of both real and synthetic urban scenes improve the robustness and generalization of learned models in 3D scene understanding?

Key findings

• The path planner proposed by Zhou et al. [46] achieves higher reconstruction quality than oblique photography while consuming less energy.
• The method by Zhang et al. [44] reduces battery consumption due to path continuity, despite slightly higher reconstruction error compared to Zhou et al. [46].
• Oblique photography, while energy-efficient, produces the roughest reconstructions due to occlusions between buildings and limited view angles.
• The integration of high-precision LiDAR scans enables robust, point-level evaluation of reconstruction accuracy and completeness across all planners.
• The simulator supports realistic simulation of drones and cars with physics-based dynamics, lighting, and weather variations, enabling diverse data generation for training and evaluation.
• The dataset provides 128k+ high-resolution images, 136 km² of urban area, and detailed instance-level annotations, forming a comprehensive benchmark for 3D urban perception and reconstruction.