[Paper Review] Depth Pro: Sharp Monocular Metric Depth in Less Than a Second

Aleksei Bochkovskii, Amaël Delaunoy+5 more|arXiv (Cornell University)|Oct 2, 2024

Advanced Measurement and Metrology Techniques14 citations

TL;DR

Depth Pro is a zero-shot metric monocular depth model that outputs high-resolution, sharp, metric depth maps with absolute scale in 0.3 seconds on a V100 GPU, using a multi-scale ViT-based architecture and a combined real/synthetic training curriculum.

ABSTRACT

We present a foundation model for zero-shot metric monocular depth estimation. Our model, Depth Pro, synthesizes high-resolution depth maps with unparalleled sharpness and high-frequency details. The predictions are metric, with absolute scale, without relying on the availability of metadata such as camera intrinsics. And the model is fast, producing a 2.25-megapixel depth map in 0.3 seconds on a standard GPU. These characteristics are enabled by a number of technical contributions, including an efficient multi-scale vision transformer for dense prediction, a training protocol that combines real and synthetic datasets to achieve high metric accuracy alongside fine boundary tracing, dedicated evaluation metrics for boundary accuracy in estimated depth maps, and state-of-the-art focal length estimation from a single image. Extensive experiments analyze specific design choices and demonstrate that Depth Pro outperforms prior work along multiple dimensions. We release code and weights at https://github.com/apple/ml-depth-pro

Motivation & Objective

• Develop a zero-shot, metric monocular depth estimator that outputs absolute-scale depth without camera intrinsics.
• Achieve high-resolution, boundary-accurate depth maps with fine structures (hair, fur, vegetation).
• Maintain low latency to enable interactive view synthesis and related applications.
• Estimate focal length from a single image to provide robust metric depth without EXIF data.
• Introduce evaluation metrics for depth-boundary fidelity using matting/segmentation datasets.

Proposed method

• Apply a plain ViT-based architecture operating at a fixed high resolution (1536x1536) by processing patches across multiple scales and fusing them with a DPT-style decoder.
• Predict canonical inverse depth C from an input image I, then compute metric depth via Dm = f_px / (w C) where f_px is the focal length and w is image width.
• Train with a two-stage curriculum mixing real and synthetic datasets to balance boundary sharpness and pixelwise accuracy (Stage 1: robust cross-domain features; Stage 2: sharpen boundaries using high-quality synthetic ground truth).
• Introduce multi-scale derivative losses (MAGE, MALE, MSGE) to enforce sharp boundaries and fine details across scales.
• Propose zero-shot focal length estimation from intermediate features plus a dedicated focal-length head trained separately to predict the horizontal field of view.
• Develop boundary-focused evaluation metrics leveraging matting/segmentation annotations to quantify occluding contours and boundary recall.

Experimental results

Research questions

• RQ1 Can a zero-shot monocular depth model produce metric, absolute-scale depth without camera intrinsics?
• RQ2 Does a multi-scale ViT-based architecture yield sharper depth boundaries at high resolution while maintaining fast runtimes?
• RQ3 How can training with a mix of real and synthetic data, plus specialized boundary-focused losses, improve boundary fidelity in depth maps?
• RQ4 Is focal length estimable from a single image in a zero-shot setting with high accuracy?
• RQ5 Do new boundary-aware evaluation metrics correlate with practical improvements in view synthesis and editing tasks?

Key findings

• Depth Pro achieves 2.25-megapixel depth maps at 0.3s on a V100 GPU with absolute metric depth and no camera intrinsics.
• Depth Pro yields superior boundary accuracy, outperforming prior work by a multiplicative margin in boundary recall across multiple datasets.
• On zero-shot metric depth, Depth Pro ranks best on average across Booster, ETH3D, Middlebury, NuScenes, Sintel, and Sun-RGBD datasets.
• Depth Pro is considerably faster and sharper in boundaries than diffusion-based Marigold and patch-based PatchFusion baselines.
• Focal length estimation from a single image with Depth Pro significantly outperforms prior focal-length predictors on a curated zero-shot dataset.