[Paper Review] GauU-Scene V2: Assessing the Reliability of Image-Based Metrics with Expansive Lidar Image Dataset Using 3DGS and NeRF

Butian Xiong, Nanjun Zheng+2 more|arXiv (Cornell University)|Apr 7, 2024

Advanced Optical Sensing Technologies4 citations

TL;DR

This paper introduces GauU-Scene V2, a large-scale multimodal dataset combining 6.67 km² of UAV-captured RGB images and 627.5 million LiDAR points, enabling rigorous evaluation of 3D scene reconstruction. Using 3D Gaussian Splatting (3DGS) and NeRF, it reveals that image-based metrics (SSIM, LPIPS, PSNR) are unreliable for geometry assessment, as they contradict geometric metrics like Chamfer distance, exposing critical flaws in 3DGS-based geometric reconstruction.

ABSTRACT

We introduce a novel, multimodal large-scale scene reconstruction benchmark that utilizes newly developed 3D representation approaches: Gaussian Splatting and Neural Radiance Fields (NeRF). Our expansive U-Scene dataset surpasses any previously existing real large-scale outdoor LiDAR and image dataset in both area and point count. GauU-Scene encompasses over 6.5 square kilometers and features a comprehensive RGB dataset coupled with LiDAR ground truth. Additionally, we are the first to propose a LiDAR and image alignment method for a drone-based dataset. Our assessment of GauU-Scene includes a detailed analysis across various novel viewpoints, employing image-based metrics such as SSIM, LPIPS, and PSNR on NeRF and Gaussian Splatting based methods. This analysis reveals contradictory results when applying geometric-based metrics like Chamfer distance. The experimental results on our multimodal dataset highlight the unreliability of current image-based metrics and reveal significant drawbacks in geometric reconstruction using the current Gaussian Splatting-based method, further illustrating the necessity of our dataset for assessing geometry reconstruction tasks. We also provide detailed supplementary information on data collection protocols and make the dataset available on the following anonymous project page

Motivation & Objective

• To address the lack of large-scale, multimodal datasets with aligned LiDAR and image data for 3D scene reconstruction evaluation.
• To investigate the reliability of image-based metrics (SSIM, LPIPS, PSNR) in assessing 3D reconstruction quality when compared to geometric metrics like Chamfer distance.
• To expose and analyze geometric reconstruction flaws in current 3D Gaussian Splatting (3DGS) methods using a high-fidelity, large-scale benchmark.
• To develop and validate a novel statistical scale matching method for aligning drone-based LiDAR point clouds with SfM camera positions in WGS 84 coordinates.
• To provide a publicly available, large-scale benchmark dataset to support future research in geometric reconstruction and multimodal fusion.

Proposed method

• The authors created GauU-Scene V2, a 6.67 km² outdoor dataset collected via UAVs using high-resolution RGB cameras and high-precision LiDAR, with over 4,600 aligned images and 627.5 million LiDAR points.
• They developed a statistical scale matching method to align LiDAR point clouds with SfM camera positions in COLMAP format, ensuring consistent coordinate systems across modalities.
• The dataset supports training and evaluation of both NeRF and 3D Gaussian Splatting (3DGS) models using both image-based and geometric-based metrics.
• Image-based metrics (SSIM, LPIPS, PSNR) were evaluated on novel views, while geometric reconstruction was assessed using Chamfer distance between predicted and ground-truth point clouds.
• The authors conducted qualitative and quantitative analysis of 3DGS, identifying nearly transparent Gaussian splats (with low alpha values) as a key source of geometric degradation.
• They further analyzed mesh outputs from SuGaR, revealing a high number of outliers that explain its poor performance in Chamfer distance despite good visual quality.

Figure 1. The dataset prepared for input into the neural field and Gaussian Splatting typically consists of camera positions and images in COLMAP format. The Structure from Motion (SfM) algorithm implemented in COLMAP initializes camera positions randomly, which may not align with LiDAR data in WGS

Experimental results

Research questions

• RQ1 How reliable are standard image-based metrics (SSIM, LPIPS, PSNR) in evaluating 3D scene reconstruction when ground truth is available in 3D point clouds?
• RQ2 Why do 3D Gaussian Splatting (3DGS) methods fail to accurately reconstruct underlying geometry despite high image fidelity?
• RQ3 To what extent do geometric reconstruction metrics like Chamfer distance contradict image-based metrics in evaluating 3D reconstruction quality?
• RQ4 What causes the high number of outliers in mesh outputs from geometric alignment methods like SuGaR, and how do they affect metric evaluation?
• RQ5 How do lighting and weather variations during image capture influence the placement and transparency of Gaussian splats in 3DGS?

Key findings

• Image-based metrics such as SSIM, LPIPS, and PSNR show contradictory results when compared to geometric metrics like Chamfer distance, indicating that image fidelity does not reliably reflect geometric accuracy.
• 3D Gaussian Splatting (3DGS) methods produce nearly transparent Gaussian splats (with very low alpha values) that float in 3D space and degrade geometric reconstruction metrics, despite contributing little to visible rendering.
• Even after removing nearly transparent splats, image quality remains largely unchanged, suggesting these splats are artifacts that harm geometric evaluation without improving visual output.
• SuGaR, a method designed for geometric alignment, ranks last in Chamfer distance due to a high number of mesh outliers, despite producing visually plausible reconstructions.
• The study reveals that current geometric metrics are also unreliable when using simple point correspondence (e.g., mean of 3DGS), indicating a need for more robust geometric evaluation frameworks.
• The dataset GauU-Scene V2, with 6.67 km² and 627.5 million LiDAR points, is the largest publicly available multimodal dataset for large-scale 3D scene reconstruction evaluation.

Figure 2. Our dataset is organized into six primary sections. The first and second scenes, located in the top row of the graph, feature the Modern Building and the Russian Building, respectively. The third and fourth scenes, depicted on the second line of the graph, represent a campus and a college.