[Paper Review] Learning to Recover 3D Scene Shape from a Single Image

Wei Yin, Jianming Zhang+5 more|arXiv (Cornell University)|Dec 17, 2020

Advanced Vision and Imaging47 references4 citations

TL;DR

This paper proposes a two-stage framework for recovering accurate 3D scene shape from a single in-the-wild image by first predicting depth up to an unknown scale and shift, then using 3D point cloud encoders to predict the missing depth shift and focal length. The method achieves state-of-the-art zero-shot generalization on nine unseen datasets using a novel image-level normalized regression loss and a pair-wise surface normal regression loss for robust training on mixed multi-source data.

ABSTRACT

Despite significant progress in monocular depth estimation in the wild, recent state-of-the-art methods cannot be used to recover accurate 3D scene shape due to an unknown depth shift induced by shift-invariant reconstruction losses used in mixed-data depth prediction training, and possible unknown camera focal length. We investigate this problem in detail, and propose a two-stage framework that first predicts depth up to an unknown scale and shift from a single monocular image, and then use 3D point cloud encoders to predict the missing depth shift and focal length that allow us to recover a realistic 3D scene shape. In addition, we propose an image-level normalized regression loss and a normal-based geometry loss to enhance depth prediction models trained on mixed datasets. We test our depth model on nine unseen datasets and achieve state-of-the-art performance on zero-shot dataset generalization. Code is available at: https://git.io/Depth

Motivation & Objective

• To address the critical problem of inaccurate 3D scene shape reconstruction from single monocular images due to unknown depth shift and focal length.
• To overcome the limitations of shift-invariant losses in existing monocular depth models that prevent metric 3D shape recovery.
• To enable zero-shot generalization to unseen datasets by leveraging 3D point cloud encoders trained on synthetic or scanned data.
• To improve monocular depth estimation on mixed, diverse datasets by introducing new training losses that enhance geometry and robustness.
• To bridge the domain gap between image-based and 3D point cloud-based priors for better 3D reconstruction fidelity.

Proposed method

• A two-stage framework: first, a CNN-based depth prediction module estimates depth maps up to an unknown scale and shift from a single image.
• Second, a 3D point cloud reconstruction module uses learned point cloud encoders to predict the missing depth shift and focal length adjustment factors.
• The depth prediction model is trained with a novel image-level normalized regression loss that transforms depth to a canonical, scale-shift-invariant space for improved training stability.
• A pair-wise surface normal regression loss is introduced to enhance geometric accuracy, particularly on edges and planar regions, with sampling strategies tailored to data quality.
• The method leverages synthetic and 3D-scanned data for training the point cloud encoders, reducing domain gap compared to image-based priors.
• An auxiliary path in the network outputs inverse depth, with ranking and normalized regression losses applied to improve generalization across diverse data sources.

Experimental results

Research questions

• RQ1 Can a fully data-driven method recover accurate 3D scene shape from a single in-the-wild image when depth shift and focal length are unknown?
• RQ2 How can shift-invariant losses in mixed-dataset training be mitigated to enable metric 3D reconstruction?
• RQ3 Can 3D point cloud encoders trained on synthetic or scanned data generalize to real-world in-the-wild scenes for shape refinement?
• RQ4 To what extent do image-level normalized regression and surface normal-based losses improve depth prediction on heterogeneous datasets?
• RQ5 Does the proposed framework achieve state-of-the-art zero-shot generalization across diverse, unseen datasets?

Key findings

• The proposed method achieves state-of-the-art performance on zero-shot transfer to nine unseen datasets, demonstrating strong generalization beyond training data.
• The use of 3D point cloud encoders for predicting depth shift and focal length significantly improves 3D shape realism and accuracy compared to standard depth prediction alone.
• The image-level normalized regression loss improves training robustness on mixed datasets by standardizing depth distributions across sources with unknown scale and shift.
• The pair-wise surface normal regression loss enhances geometric fidelity, especially in edge and planar regions, reducing artifacts in reconstructed 3D shapes.
• The framework successfully recovers realistic 3D scene shapes from single images across diverse in-the-wild scenes, as validated by qualitative results on unseen data.
• The method generalizes well to low-quality and high-quality datasets alike, showing resilience to variations in depth supervision quality.