[Paper Review] Single-Image Depth Perception in the Wild

Weifeng Chen, Fu Zhao+2 more|arXiv (Cornell University)|Apr 13, 2016

Advanced Vision and Imaging27 references98 citations

TL;DR

This work introduces the Depth in the Wild (DIW) dataset with relative-depth annotations and presents a single-end-to-end deep network that predicts metric depth from a single image using relative-depth supervision, achieving state-of-the-art performance on wild images.

ABSTRACT

This paper studies single-image depth perception in the wild, i.e., recovering depth from a single image taken in unconstrained settings. We introduce a new dataset "Depth in the Wild" consisting of images in the wild annotated with relative depth between pairs of random points. We also propose a new algorithm that learns to estimate metric depth using annotations of relative depth. Compared to the state of the art, our algorithm is simpler and performs better. Experiments show that our algorithm, combined with existing RGB-D data and our new relative depth annotations, significantly improves single-image depth perception in the wild.

Motivation & Objective

• Motivate depth estimation from a single image in unconstrained, real-world settings (in the wild).
• Create a large-scale dataset (Depth in the Wild) with relative-depth annotations for diverse images.
• Develop an end-to-end deep network trained with relative-depth supervision to predict metric depth.
• Demonstrate that combining DIW with existing RGB-D data improves wild-depth estimation.

Proposed method

• Crowdsource relative-depth annotations on 495K diverse wild images, sampling one point-pair per image.
• Propose an end-to-end deep network that outputs pixel-wise metric depth, trained with a relative-depth loss.
• Use a multi-scale hourglass-style architecture with skip connections to propagate information across scales.
• Train with a ranking-like loss that enforces correct ordering and equality for depth at queried point pairs.
• Evaluate on NYU Depth (indoor) and the DIW dataset (in the wild) to show improvements over prior methods.

Experimental results

Research questions

• RQ1 Can relative-depth annotations on wild images be used to learn accurate metric depth from a single image?
• RQ2 Does an end-to-end network trained with relative-depth loss outperform ordinal-relation based methods and RGB-D supervised baselines in the wild?
• RQ3 How does pre-training on RGB-D data and fine-tuning on DIW affect performance on diverse scenes (indoor and outdoor)?

Key findings

• The proposed approach yields pixel-wise depth with better ordinal consistency than prior methods.
• On NYU Depth, the method outperforms Zoran et al. (ordinal relations) and approaches Eigen’s depth estimation trained on full metric data.
• On Depth in the Wild, pre-training on NYU Depth and fine-tuning on DIW provides the best WHDR scores, showing strong gains in outdoor and indoor scenes.
• Combining existing RGB-D data with DIW annotations significantly improves single-image depth perception in the wild.
• Training with random point pairs (instead of superpixels) can match or exceed performance when enough pairs per image are used.