[Paper Review] Depth Map Prediction from a Single Image using a Multi-Scale Deep Network

David Eigen, Christian Puhrsch+1 more|arXiv (Cornell University)|Jun 9, 2014

Advanced Vision and Imaging16 references2,258 citations

TL;DR

This paper proposes a multi-scale deep network with two stacked components: a global coarse predictor and a local refinement network, to predict depth maps from single images. By using scale-invariant loss and leveraging large-scale raw datasets, it achieves state-of-the-art performance on NYU Depth and KITTI benchmarks, accurately capturing depth boundaries without superpixelation.

ABSTRACT

Predicting depth is an essential component in understanding the 3D geometry of a scene. While for stereo images local correspondence suffices for estimation, finding depth relations from a single image is less straightforward, requiring in-tegration of both global and local information from various cues. Moreover, the task is inherently ambiguous, with a large source of uncertainty coming from the overall scale. In this paper, we present a new method that addresses this task by employing two deep network stacks: one that makes a coarse global prediction based on the entire image, and another that refines this prediction locally. We also apply a scale-invariant error to help measure depth relations rather than scale. By leveraging the raw datasets as large sources of training data, our method achieves state-of-the-art results on both NYU Depth and KITTI, and matches detailed depth boundaries without the need for superpixelation. 1

Motivation & Objective

• Address the inherent ambiguity in single-image depth estimation, particularly scale uncertainty.
• Integrate both global context and local details to improve depth map accuracy.
• Overcome limitations of existing methods that rely on superpixelation or fail to preserve fine depth boundaries.
• Develop a robust method that generalizes well across diverse scenes and datasets.
• Achieve state-of-the-art performance on benchmark datasets like NYU Depth and KITTI without requiring multi-view supervision.

Proposed method

• Employ a two-stack deep network architecture: a global encoder for coarse depth prediction using the full image context.
• Use a refinement network that operates locally to enhance detail and correct coarse predictions.
• Apply a scale-invariant loss function to prioritize accurate depth relations over absolute scale, reducing scale ambiguity.
• Train the entire network end-to-end using large-scale raw datasets as supervision.
• Leverage multi-scale features to capture both global scene structure and local depth variations.
• Avoid superpixelation by directly predicting dense depth maps with high boundary fidelity.

Experimental results

Research questions

• RQ1 Can a multi-scale deep network architecture effectively combine global and local cues for single-image depth prediction?
• RQ2 Does using a scale-invariant loss function improve depth estimation accuracy by reducing scale ambiguity?
• RQ3 Can the proposed method achieve state-of-the-art performance without relying on superpixelation or multi-view data?
• RQ4 How well does the method generalize across diverse indoor and outdoor scenes in benchmark datasets?
• RQ5 To what extent does the two-stage network design (coarse + refinement) improve boundary accuracy compared to single-stage models?

Key findings

• The proposed method achieves state-of-the-art performance on the NYU Depth dataset, outperforming prior approaches in depth estimation accuracy.
• On the KITTI benchmark, the method matches or exceeds the performance of existing SOTA methods in depth prediction quality.
• The model successfully preserves detailed depth boundaries without the need for superpixelation, improving localization accuracy.
• The use of scale-invariant loss significantly reduces scale-related errors, enhancing generalization across scenes with varying depth ranges.
• The two-stage network design—coarse global prediction followed by local refinement—leads to more accurate and consistent depth maps.
• End-to-end training on large-scale raw datasets enables strong generalization and robustness across diverse real-world scenes.