[Paper Review] MVDepthNet: Real-time Multiview Depth Estimation Neural Network

Kaixuan Wang, Shaojie Shen|arXiv (Cornell University)|Jul 23, 2018

Advanced Vision and Imaging3 references4 citations

TL;DR

MVDepthNet is a real-time convolutional neural network for multiview depth estimation that constructs a cost volume from multiple image-pose pairs to enable efficient, geometry-aware depth prediction regardless of camera parameters or number of views. It achieves state-of-the-art accuracy in real-time monocular dense mapping, outperforming methods like DeMoN and FCRN on both indoor and outdoor datasets with improved handling of textureless and reflective regions.

ABSTRACT

Although deep neural networks have been widely applied to computer vision problems, extending them into multiview depth estimation is non-trivial. In this paper, we present MVDepthNet, a convolutional network to solve the depth estimation problem given several image-pose pairs from a localized monocular camera in neighbor viewpoints. Multiview observations are encoded in a cost volume and then combined with the reference image to estimate the depth map using an encoder-decoder network. By encoding the information from multiview observations into the cost volume, our method achieves real-time performance and the flexibility of traditional methods that can be applied regardless of the camera intrinsic parameters and the number of images. Geometric data augmentation is used to train MVDepthNet. We further apply MVDepthNet in a monocular dense mapping system that continuously estimates depth maps using a single localized moving camera. Experiments show that our method can generate depth maps efficiently and precisely.

Motivation & Objective

• To develop a real-time, end-to-end trainable neural network for depth estimation from multiple monocular views with known camera poses.
• To overcome challenges in multiview depth estimation such as arbitrary epipolar lines and parameterized triangulation in monocular systems.
• To enable generalization across varying camera intrinsics, numbers of views, and scene types without architectural retraining.
• To improve robustness to textureless regions, reflective surfaces, and long-range depth estimation beyond RGB-D limitations.
• To integrate the network into a real-time monocular dense mapping system for continuous depth estimation.

Proposed method

• MVDepthNet constructs a cost volume by encoding multiview observations from input image-pose pairs, representing potential depth hypotheses across views.
• The cost volume is processed by an encoder-decoder network to predict depth maps, avoiding explicit pixel matching across views.
• Geometric data augmentation is applied to the cost volume and ground truth depth, preserving per-pixel consistency during training.
• Camera poses and intrinsic parameters are encoded into the cost volume to enable triangulation without fixed epipolar constraints.
• The network is trained end-to-end using supervised loss on depth maps, with augmentation to improve generalization on limited real-world data.
• The method is integrated into a monocular dense mapping system that continuously estimates depth using a moving camera.

Experimental results

Research questions

• RQ1 Can a deep learning model achieve real-time depth estimation from multiple monocular views with known poses, without requiring stereo calibration?
• RQ2 How can multiview geometric constraints be effectively encoded in a neural network for robust depth prediction across varying camera baselines and intrinsic parameters?
• RQ3 Can geometric data augmentation preserve per-pixel consistency and improve generalization when training on limited real-world datasets?
• RQ4 How does the proposed method compare in accuracy and robustness to RGB-D cameras and state-of-the-art monocular depth estimation methods on challenging scenes?
• RQ5 Can the network generalize to textureless, reflective, and distant objects where RGB-D sensors fail?

Key findings

• MVDepthNet achieves real-time performance on standard hardware, enabling continuous depth estimation in monocular dense mapping systems.
• On the TUM RGB-D dataset, MVDepthNet achieves a mean L1-rel error of 0.109 on the desk sequence, outperforming VI-MEAN (0.623) and REMODE (1.314).
• In the ICL-NUIM dataset, MVDepthNet achieves a mean L1-rel error of 0.076 on the xyz sequence, outperforming VI-MEAN (0.564) and REMODE (0.801).
• MVDepthNet successfully estimates depth in textureless and reflective regions, as shown in qualitative results where RGB-D cameras fail, marked by red circles in Figure 1.
• The method achieves 100% valid depth estimation density on all tested sequences, while VI-MEAN and REMODE show significantly lower density due to failure in textureless or motion-blurred regions.
• MVDepthNet reduces streaking artifacts common in VI-MEAN, which uses 1D regularization, and handles occlusions better than DeepMVS, which relies on pixel classification loss.