[Paper Review] Semi-Supervised Deep Learning for Monocular Depth Map Prediction

Yevhen Kuznietsov, Jörg Stückler+1 more|arXiv (Cornell University)|Feb 9, 2017

Advanced Vision and ImagingComputer Science22 references90 citations

TL;DR

The paper presents a semi-supervised approach that combines sparse LiDAR depth supervision with unsupervised stereo image alignment losses to train a deep residual encoder-decoder for monocular depth estimation, achieving state-of-the-art results on KITTI.

ABSTRACT

Supervised deep learning often suffers from the lack of sufficient training data. Specifically in the context of monocular depth map prediction, it is barely possible to determine dense ground truth depth images in realistic dynamic outdoor environments. When using LiDAR sensors, for instance, noise is present in the distance measurements, the calibration between sensors cannot be perfect, and the measurements are typically much sparser than the camera images. In this paper, we propose a novel approach to depth map prediction from monocular images that learns in a semi-supervised way. While we use sparse ground-truth depth for supervised learning, we also enforce our deep network to produce photoconsistent dense depth maps in a stereo setup using a direct image alignment loss. In experiments we demonstrate superior performance in depth map prediction from single images compared to the state-of-the-art methods.

Motivation & Objective

• Address the lack of dense ground-truth depth data for outdoor scenes.
• Leverage sparse LiDAR measurements alongside unsupervised stereo cues.
• Develop a semi-supervised loss that combines supervised, unsupervised, and regularization terms.
• Utilize a deep residual encoder-decoder with long skip connections for high-detail depth maps.
• Demonstrate state-of-the-art performance on the KITTI dataset and analyze ablations.

Proposed method

• Predict inverse depth at each pixel from a monocular RGB image using a ResNet-50 based encoder-decoder with long skip connections.
• Train with a unified semi-supervised loss that includes: (i) supervised depth residuals projected from sparse LiDAR, (ii) unsupervised photometric (image alignment) loss across stereo pairs, (iii) a depth smoothness regularization term.
• Use berHu loss for the supervised term with adaptive delta to emphasize larger residuals.
• Enforce symmetric photometric consistency between left and right views without explicit left-right disparity constraints.
• Initialize with ImageNet-pretrained encoder, progressively fade-in the supervised term to aid convergence.
• Evaluate on KITTI using standard metrics (RMSE, RMSE log, accuracy at thresholds, ARD, SRD) and compare with state-of-the-art.

Experimental results

Research questions

• RQ1 Can sparse ground-truth depth from LiDAR be effectively combined with unsupervised stereo photometric losses to train accurate monocular depth predictors?
• RQ2 Does a semi-supervised framework improve depth accuracy and convergence speed compared to purely supervised or unsupervised approaches?
• RQ3 What is the impact of architectural choices (long skip connections, berHu loss, symmetric losses) on depth map quality?
• RQ4 How well does the model generalize to other outdoor datasets beyond KITTI?

Key findings

• The proposed semi-supervised approach achieves state-of-the-art depth prediction on KITTI, outperforming prior methods on multiple metrics.
• Using the unsupervised image alignment loss improves performance, especially for far distances where ground-truth is sparse.
• BerHu loss in the supervised term yields cleaner, less noisy depth maps than L2 loss.
• Long skip connections and Gaussian smoothing in the unsupervised term contribute to faster convergence and better detail in depth maps.
• The fully semi-supervised model (Ours) achieves RMSE 4.621, RMSE log 0.189, delta<1.25 0.862, delta<1.25^2 0.960, delta<1.25^3 0.986 on 0-80 m KITTI test (cap 80 m).
• Purely supervised variant performs worse (RMSE 4.815, RMSE log 0.194, delta<1.25 0.845, delta<1.25^2 0.957, delta<1.25^3 0.987).
• Purely unsupervised variant underperforms on KITTI with cap 0-80 m (RMSE 8.700, RMSE log 0.367, delta<1.25 0.752, delta<1.25^2 0.904, delta<1.25^3 0.952).