[Paper Review] MVSNet: Depth Inference for Unstructured Multi-view Stereo

Yao Yao, Zixin Luo+3 more|arXiv (Cornell University)|Apr 7, 2018

Advanced Vision and Imaging29 references61 citations

TL;DR

MVSNet presents an end-to-end deep learning approach for per-view depth map inference from unstructured multi-view images using differentiable homography for cost volume construction, 3D CNN regularization, and depth refinement.

ABSTRACT

We present an end-to-end deep learning architecture for depth map inference from multi-view images. In the network, we first extract deep visual image features, and then build the 3D cost volume upon the reference camera frustum via the differentiable homography warping. Next, we apply 3D convolutions to regularize and regress the initial depth map, which is then refined with the reference image to generate the final output. Our framework flexibly adapts arbitrary N-view inputs using a variance-based cost metric that maps multiple features into one cost feature. The proposed MVSNet is demonstrated on the large-scale indoor DTU dataset. With simple post-processing, our method not only significantly outperforms previous state-of-the-arts, but also is several times faster in runtime. We also evaluate MVSNet on the complex outdoor Tanks and Temples dataset, where our method ranks first before April 18, 2018 without any fine-tuning, showing the strong generalization ability of MVSNet.

Motivation & Objective

• Motivate improved depth map inference in MVS with unstructured views.
• Propose an end-to-end network that builds a 3D cost volume from 2D features using differentiable homography.
• Enable flexible handling of arbitrary N-view inputs through a variance-based cost metric.
• Regress and refine per-view depth maps, enabling efficient large-scale reconstructions.

Proposed method

• Extract multi-scale 2D image features with a shared 8-layer CNN.
• Construct a 3D cost volume on the reference camera frustum using differentiable homography warping.
• Aggregate N view volumes with a variance-based cost metric to form the final cost volume.
• Regularize the cost volume with a multi-scale 3D CNN and produce a probability volume via softmax along depth.
• Compute initial depth as a differentiable soft argmin over depth hypotheses.
• Refine the initial depth with a guidance network using the reference image to produce a final depth map.
• Train with a loss combining initial and refined depth map errors on valid ground-truth pixels.

Experimental results

Research questions

• RQ1 Can unstructured multi-view inputs be effectively integrated into a per-view depth map inference framework using a differentiable projection mechanism?
• RQ2 Does a variance-based multi-view cost metric improve depth estimation accuracy and robustness compared to mean-based approaches?
• RQ3 How does per-view depth map refinement leveraging the reference image affect boundary accuracy and overall depth quality?
• RQ4 What is the generalization capability of the method from indoor DTU data to outdoor Tanks and Temples data without fine-tuning?

Key findings

• Outperforms prior methods on DTU in completeness and overall quality, with strong robustness in textureless and reflective regions.
• Achieves state-of-the-art results on Tanks and Temples without fine-tuning, demonstrating strong generalization.
• Variance-based cost metric leads to faster convergence and better validation performance than mean-based alternatives.
• Depth map refinement improves boundary accuracy and overall depth quality.
• Runtime is significantly faster than prior methods (roughly 230 seconds per scan, ~4.7 seconds per view).
• The model supports varying numbers of input views (N) and maintains improved performance when more views are used.