[Paper Review] A metric for evaluating 3D reconstruction and mapping performance with no ground truthing

Guoxiang Zhang, YangQuan Chen|arXiv (Cornell University)|Jan 25, 2021

Robotics and Sensor-Based Localization16 references4 citations

TL;DR

This paper proposes Dense Map Posterior (DMP), a novel metric for evaluating 3D reconstruction and SLAM performance without requiring ground truth data. By computing a log-likelihood ratio based on sensor observations and reconstructed maps, DMP achieves strong correlation with ground truth metrics like SMD and trajectory RMSE, enabling reliable, self-supervised evaluation and loop detection filtering.

ABSTRACT

It is not easy when evaluating 3D mapping performance because existing metrics require ground truth data that can only be collected with special instruments. In this paper, we propose a metric, dense map posterior (DMP), for this evaluation. It can work without any ground truth data. Instead, it calculates a comparable value, reflecting a map posterior probability, from dense point cloud observations. In our experiments, the proposed DMP is benchmarked against ground truth-based metrics. Results show that DMP can provide a similar evaluation capability. The proposed metric makes evaluating different methods more flexible and opens many new possibilities, such as self-supervised methods and more available datasets.

Motivation & Objective

• Address the challenge of evaluating 3D reconstruction and SLAM systems without expensive, hard-to-acquire ground truth data such as LiDAR or motion capture trajectories.
• Develop a metric that enables reliable performance evaluation using only sensor observations and reconstructed maps, avoiding reliance on external ground truth.
• Provide a scalable, self-supervised evaluation framework to support large-scale real-world dataset creation and self-supervised learning in 3D mapping.
• Enable robust loop closure evaluation by ranking loop candidates based on map consistency, reducing false positives in SLAM pipelines.

Proposed method

• Formulate the dense map posterior (DMP) as a Bayesian estimate of map consistency given sensor observations, using Bayes’ rule to derive a computable likelihood ratio.
• Compute the log-likelihood ratio (LLR) between two map hypotheses using the product of Gaussian likelihoods for each 3D point or depth pixel, assuming zero-mean error and known variance.
• Use sensor models (e.g., camera projection or LiDAR range-bearing models) to predict expected observations from a reconstructed map and trajectory.
• Define the metric as the negative log-likelihood, r(M, Z) = -log p(Z|M), which serves as a score: lower values indicate better map consistency.
• Enable parallel computation using CUDA and OpenGL for efficient evaluation on modern GPUs, achieving sub-5s evaluation times per dataset.
• Apply the metric to rank loop closures by evaluating the consistency of fused surfel models after adding each candidate loop.

Experimental results

Research questions

• RQ1 Can a self-supervised metric be developed to evaluate 3D reconstruction and SLAM performance without requiring ground truth data?
• RQ2 How well does the proposed DMP correlate with established ground truth-based metrics such as SMD and trajectory RMSE?
• RQ3 Can DMP effectively rank and filter false-positive loop closures in SLAM systems?
• RQ4 In what scenarios does DMP fail to align with ground truth metrics, and can these be mitigated?

Key findings

• DMP shows strong positive correlation with ground truth-based metrics: the correlation coefficient with SMD is 0.93 and with trajectory RMSE is 0.91 across tested datasets.
• The metric successfully identifies poor reconstructions—e.g., a collapsed model with DMP = 2254.0, while a high-quality reconstruction scores 14.9.
• In loop closure evaluation, DMP achieves high-precision ranking, with precision remaining above 90% until recall reaches 100%, indicating effective false-positive filtering.
• The method is computationally efficient, evaluating a TUM dataset in 2.7 seconds and an ICL-NUIM dataset in 4.2 seconds on an NVIDIA Titan X Pascal.
• Outliers in evaluation occur primarily in non-loopy sequences with sparse coverage; adding overlapping scans resolves the issue.
• DMP enables self-supervised learning and large-scale real-world dataset creation by removing dependency on ground truth data.