[Paper Review] Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image

Fangchang Ma, Sertaç Karaman|arXiv (Cornell University)|Sep 21, 2017

Advanced Vision and ImagingComputer Science20 references100 citations

TL;DR

This paper presents a single deep regression network that predicts dense depth from a combination of an RGB image and a sparse set of depth samples, achieving substantial accuracy gains over RGB-only methods on NYU-Depth-v2 and KITTI by incorporating as few as 100 depth samples.

ABSTRACT

We consider the problem of dense depth prediction from a sparse set of depth measurements and a single RGB image. Since depth estimation from monocular images alone is inherently ambiguous and unreliable, to attain a higher level of robustness and accuracy, we introduce additional sparse depth samples, which are either acquired with a low-resolution depth sensor or computed via visual Simultaneous Localization and Mapping (SLAM) algorithms. We propose the use of a single deep regression network to learn directly from the RGB-D raw data, and explore the impact of number of depth samples on prediction accuracy. Our experiments show that, compared to using only RGB images, the addition of 100 spatially random depth samples reduces the prediction root-mean-square error by 50% on the NYU-Depth-v2 indoor dataset. It also boosts the percentage of reliable prediction from 59% to 92% on the KITTI dataset. We demonstrate two applications of the proposed algorithm: a plug-in module in SLAM to convert sparse maps to dense maps, and super-resolution for LiDARs. Software and video demonstration are publicly available.

Motivation & Objective

• Motivate robust depth estimation by fusing RGB with sparse depth samples from low-res sensors or SLAM outputs.
• Propose a single CNN architecture that ingests RGB-D data (RGB plus sparse depth) to predict dense depth.
• Evaluate how the number of depth samples affects prediction accuracy on indoor (NYU-Depth-v2) and outdoor (KITTI) datasets.
• Demonstrate practical applications in SLAM/VIO dense map augmentation and LiDAR super-resolution.

Proposed method

• Use a CNN architecture with ResNet-based encoders (ResNet-18 for KITTI, ResNet-50 for NYU-Depth-v2) and a 4-upsampling decoder (UpProj).
• Train with online sampled sparse depth on ground truth using a Bernoulli scheme with probability p = m/n, where m is target sample count and n is total valid depth pixels.
• Apply online data augmentation (scaling, rotation, color jitter, normalization, flips) with nearest-neighbor interpolation to preserve sparse points.
• Optimize with L1 loss as default (relatively robust to outliers and preserves edges).
• Compare different upsampling modules (DeConv, UpConv, UpProj) and first-layer convolutions (Conv, DepthWise, ChanDrop) to optimize performance.

Experimental results

Research questions

• RQ1 How does adding sparse depth samples to an RGB input improve dense depth prediction compared to RGB alone?
• RQ2 What is the impact of the number of sparse depth samples on prediction accuracy for indoor and outdoor datasets?
• RQ3 Can the RGB+sparse-depth model serve as a plug-in for SLAM/VIO to generate dense maps and enable LiDAR super-resolution?
• RQ4 Which network design choices (encoder type, upsampling method, initial convolution) yield best depth prediction performance?

Key findings

• In NYU-Depth-v2, adding 100 sparse depth samples reduces RMSE by about 50% compared to RGB only.
• On KITTI, 100 sparse depth samples increase reliable predictions from 59% to 92%.
• RGBd (RGB plus ~100 sparse depth samples) significantly outperforms RGB or sparse-depth alone, and 200–1000 samples continue to improve until saturation.
• For NYU-Depth-v2, RGBd with 100 samples achieves RMSE ~0.25 m and REL ~0.05, substantially better than RGB-based methods with no depth inputs.
• For KITTI, RGBd with 100 samples achieves RMSE ~3.5 m and REL ~0.07, outperforming RGB and some fusion methods with substantially fewer depth samples.
• The method enables dense map reconstruction from sparse SLAM/VIO landmarks and LiDAR super-resolution while using comparatively few depth inputs.