[Paper Review] From 2D to 3D: Re-thinking Benchmarking of Monocular Depth Prediction

Evin Pınar Örnek, Shristi Mudgal+4 more|arXiv (Cornell University)|Mar 15, 2022

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes a 3D-aware evaluation framework for monocular depth prediction (MDP) that shifts focus from 2D metrics to 3D geometry quality using novel metrics like F-score and a new benchmark, RIO-D3D, featuring high-quality, complete 3D depth maps from RGB-D reconstructions. The key contribution is demonstrating that 2D metrics like absrel fail to reflect true 3D structural accuracy, while 3D metrics such as F-score better align with qualitative performance and improve benchmarking for MDP and 3D scene completion.

ABSTRACT

There have been numerous recently proposed methods for monocular depth prediction (MDP) coupled with the equally rapid evolution of benchmarking tools. However, we argue that MDP is currently witnessing benchmark over-fitting and relying on metrics that are only partially helpful to gauge the usefulness of the predictions for 3D applications. This limits the design and development of novel methods that are truly aware of - and improving towards estimating - the 3D structure of the scene rather than optimizing 2D-based distances. In this work, we aim to bring structural awareness to MDP, an inherently 3D task, by exhibiting the limits of evaluation metrics towards assessing the quality of the 3D geometry. We propose a set of metrics well suited to evaluate the 3D geometry of MDP approaches and a novel indoor benchmark, RIO-D3D, crucial for the proposed evaluation methodology. Our benchmark is based on a real-world dataset featuring high-quality rendered depth maps obtained from RGB-D reconstructions. We further demonstrate this to help benchmark the closely-tied task of 3D scene completion.

Motivation & Objective

• Address the growing issue of benchmark overfitting in monocular depth prediction (MDP) due to reliance on 2D-based metrics that do not reflect true 3D structural accuracy.
• Expose the limitations of current 2D metrics (e.g., absrel) in capturing the geometric fidelity of depth predictions when back-projected into 3D space.
• Propose a new evaluation protocol based on 3D metrics—especially F-score—that better quantify structural quality and alignment with ground truth 3D geometry.
• Introduce RIO-D3D, a large-scale, real-world indoor benchmark with high-quality, denoised, complete 3D depth maps, normal, occlusion, and semantic annotations for MDP and 3D scene completion.
• Enable more accurate benchmarking of 3D scene completion by providing full 3D meshes and partial depth inputs derived from the same reconstructions.

Proposed method

• Design a 3D evaluation protocol using metrics such as chamfer distance (CD), Earth Mover’s Distance (EMD), intersection over union (IoU), and F-score to assess 3D structural quality of depth predictions.
• Develop RIO-D3D, a large-scale indoor dataset derived from high-resolution RGB-D reconstructions, ensuring complete, denoised, and accurate 3D depth ground truth.
• Use point cloud-based IoU and F-score as primary evaluation metrics due to their robustness to k-nearest neighbor (knn) matching errors and better sensitivity to structural fidelity.
• Compare state-of-the-art MDP and 3D scene completion models on RIO-D3D using both 2D and 3D metrics to demonstrate the disconnect between 2D scores and 3D structural quality.
• Integrate semantic annotations and occlusion masks into RIO-D3D to support semantic-aware depth prediction and scene completion.
• Release the RIO-D3D dataset and evaluation code publicly to enable reproducibility and community adoption of 3D-aware benchmarking.

Experimental results

Research questions

• RQ1 To what extent do commonly used 2D metrics like absrel correlate with the actual 3D geometric structure of predicted depth maps?
• RQ2 Which 3D metrics—CD, EMD, IoU, or F-score—best reflect the structural accuracy and robustness of depth predictions in real-world 3D scenes?
• RQ3 How does the quality of ground truth depth data (e.g., noise, completeness) affect the reliability of MDP benchmarking and model generalization?
• RQ4 Can a new benchmark with high-quality, complete 3D depth maps improve the evaluation and development of 3D-aware MDP and 3D scene completion methods?
• RQ5 How do state-of-the-art MDP and 3D scene completion models perform when evaluated with 3D metrics versus traditional 2D metrics on RIO-D3D?

Key findings

• 2D metrics like absrel show low variance even when depth predictions have poor 3D structure, indicating they fail to detect structural inaccuracies.
• F-score consistently outperforms CD and EMD in reflecting structural quality, as it combines precision and recall and is robust to knn matching errors.
• The F-score is more reliable than IoU and EMD, as it avoids sudden score spikes and better quantifies the percentage of correctly reconstructed 3D points.
• On RIO-D3D, voxel-based scene completion models (e.g., ForkNet) outperform point-based models (e.g., GRNet, MSN) in both CD and EMD, indicating architectural impact on 3D fidelity.
• State-of-the-art MDP models with high absrel scores often produce poor 3D structures when back-projected, highlighting the disconnect between 2D optimization and 3D utility.
• The proposed F-score metric shows greater variance across models with similar absrel values, making it a more informative indicator of 3D structural quality than 2D metrics.