[Paper Review] Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue

Ravi Garg, Vijay Kumar Bg+2 more|arXiv (Cornell University)|Mar 16, 2016

Advanced Vision and Imaging27 references319 citations

TL;DR

The paper presents a fully unsupervised CNN that learns to predict a single-view depth map from a left stereo image by reconstructing the left image from the right image using a predicted disparity, enabling end-to-end training without ground-truth depths.

ABSTRACT

A significant weakness of most current deep Convolutional Neural Networks is the need to train them using vast amounts of manu- ally labelled data. In this work we propose a unsupervised framework to learn a deep convolutional neural network for single view depth predic- tion, without requiring a pre-training stage or annotated ground truth depths. We achieve this by training the network in a manner analogous to an autoencoder. At training time we consider a pair of images, source and target, with small, known camera motion between the two such as a stereo pair. We train the convolutional encoder for the task of predicting the depth map for the source image. To do so, we explicitly generate an inverse warp of the target image using the predicted depth and known inter-view displacement, to reconstruct the source image; the photomet- ric error in the reconstruction is the reconstruction loss for the encoder. The acquisition of this training data is considerably simpler than for equivalent systems, requiring no manual annotation, nor calibration of depth sensor to camera. We show that our network trained on less than half of the KITTI dataset (without any further augmentation) gives com- parable performance to that of the state of art supervised methods for single view depth estimation.

Motivation & Objective

• Motivate unsupervised learning for single-view depth estimation to avoid costly depth annotations.
• Propose a stereo-based autoencoder where the CNN predicts depth maps used to warp the right image back to the left for reconstruction.
• Demonstrate end-to-end training from scratch on KITTI without ground-truth depth data.
• Show coarse-to-fine training with skip connections improves depth prediction quality.
• Evaluate against state-of-the-art supervised methods and analyze data augmentation and fine-tuning benefits.

Proposed method

• Use a stereo pair with known camera motion to train a CNN that predicts a depth map for the source (left) image.
• Generate a backward warp of the right image using predicted depth and known disparity to reconstruct the left image; optimize a photometric reconstruction loss.
• Apply a simple smoothness prior on disparities to address the aperture problem.
• Adopt a coarse-to-fine architecture with skip connections to refine depth predictions across resolutions.
• Linearize the warp via Taylor expansion to enable back-propagation and perform iterative refinement across multiple training stages.
• Train with multi-stage upsampling (L7 to L12) and later fine-tune with data augmentation (color, scale, flips).

Experimental results

Research questions

• RQ1 Can a CNN be trained end-to-end from scratch in an unsupervised manner to predict depth from a single view using stereo geometry?
• RQ2 Does a photometric reconstruction loss from autoencoder-like warping provide competitive depth predictions without ground-truth depths?
• RQ3 What is the impact of coarse-to-fine training and skip connections on depth accuracy in an unsupervised setting?
• RQ4 How does the unsupervised approach compare to supervised single-view depth methods and stereo-based baselines on KITTI?
• RQ5 Can data augmentation and fine-tuning improve unsupervised single-view depth estimation performance?

Key findings

• The unsupervised CNN trained on stereo pairs achieves competitive depth predictions compared with state-of-the-art supervised methods on KITTI.
• Coarse-to-fine training with skip connections yields progressively better depth maps, especially at higher resolutions.
• Data augmentation and post-hoc fine-tuning further improve edge localization and overall depth accuracy.
• The method, being fully unsupervised and initialization-free, can approach supervised performance without ground-truth depths.
• Compared to stereo-to-CNN baselines, the autoencoder approach avoids learning biases from proxy ground-truth disparities and reduces depth errors near object edges.