[Paper Review] GauU-Scene: A Scene Reconstruction Benchmark on Large Scale 3D Reconstruction Dataset Using Gaussian Splatting

Butian Xiong, Zhuo Li+1 more|arXiv (Cornell University)|Jan 25, 2024

Remote Sensing and LiDAR Applications5 citations

TL;DR

The paper introduces GauU-Scene, a large-scale drone-LiDAR dataset (>1.5 km2) and a Gaussian Splatting baseline, plus a Lidar-Image fusion method to improve 3D scene reconstruction.

ABSTRACT

We introduce a novel large-scale scene reconstruction benchmark using the newly developed 3D representation approach, Gaussian Splatting, on our expansive U-Scene dataset. U-Scene encompasses over one and a half square kilometres, featuring a comprehensive RGB dataset coupled with LiDAR ground truth. For data acquisition, we employed the Matrix 300 drone equipped with the high-accuracy Zenmuse L1 LiDAR, enabling precise rooftop data collection. This dataset, offers a unique blend of urban and academic environments for advanced spatial analysis convers more than 1.5 km$^2$. Our evaluation of U-Scene with Gaussian Splatting includes a detailed analysis across various novel viewpoints. We also juxtapose these results with those derived from our accurate point cloud dataset, highlighting significant differences that underscore the importance of combine multi-modal information

Motivation & Objective

• Provide a large-scale, rooftop-inclusive 3D reconstruction dataset (>1.5 km^2) with high-precision LiDAR ground truth.
• Benchmark Gaussian Splatting on drone-collected data for city-scale scenes.
• Propose a LiDAR-Image fusion approach to enhance Gaussian Splatting prior and reconstruction accuracy.
• Highlight gaps between drone-based large-scale reconstruction and ground-truth point clouds to guide future work.

Proposed method

• Introduce GauU-Scene, a dataset over 1.5 km^2 assembled with DJI Matrix 300 and Zenmuse L1 LiDAR catering rooftop and urban views.
• Align coordinates by building a sparse SfM point cloud with COLMAP, then register and transform raw LiDAR points via global matching and ICP.
• Represent 3D scenes as Gaussian splats with attributes <m, σ, α, h>, enabling rendering and view synthesis.
• Fuse LiDAR priors with Gaussian Splatting by subsampling the LiDAR data and integrating alongside image-based priors.
• Evaluate Vanilla Gaussian Splatting and LiDAR-Fused Gaussian Splatting using L1 and PSNR metrics on both image and point-cloud grounds truth.
• Provide a simple pipeline to convert drone-collected data into a usable Gaussian Splatting representation.

Figure 1 : Our dataset is divided into three main parts. The first part is the top part of this graph. We call it CUHKSZ(The Chinese University of Hong Kong, Shenzhen) lower campus, and the bottom left corner shows the upper campus of CUHKSZ, and the bottom right corner shows the SMBU(Shenzhen MSU-B

Experimental results

Research questions

• RQ1 How does Gaussian Splatting perform on large-scale drone-collected scenes with rooftop and urban data?
• RQ2 What is the impact of incorporating LiDAR priors on Gaussian Splatting reconstruction quality compared to image-only priors?
• RQ3 How do ground-truth LiDAR point clouds versus image-ground-truth representations affect evaluation of 3D reconstructions?
• RQ4 What are the practical challenges in aligning drone-based RGB data with LiDAR scans for large-scale scenes?

Key findings

• The UAV-LiDAR GauU-Scene dataset covers over 1.5 km^2 and includes high-accuracy ground truth for evaluation.
• Lidar-Fused Gaussian Splatting generally yields higher accuracy than Vanilla Gaussian Splatting in point-cloud-based 3D representations, though image-based metrics show only modest gains.
• Quantitative results show improvements in ground-truth alignment when using LiDAR priors for 3D reconstruction (lower L1 losses in some configurations).
• Ground-truth point clouds provide more reliable 3D structure than image ground truth alone, underscoring the value of multi-modal data fusion.
• Edge artifacts and scale alignment remain challenges in large-scale Gaussian Splatting reconstructions, indicating areas for future refinement.

Figure 2 : The current point cloud registration method usually cannot handle different scales, so we first scale the raw point cloud to the same size as the SfM sparse point cloud. To do this, we find the maximum distance or variance in the SfM, as there are always some points far from the center in