[Paper Review] MVS2D: Efficient Multi-view Stereo via Attention-Driven 2D Convolutions

Zhenpei Yang, Zhile Ren+2 more|arXiv (Cornell University)|Apr 27, 2021

Advanced Vision and Imaging41 references4 citations

TL;DR

MVS2D proposes a highly efficient multi-view stereo method that leverages an attention mechanism to integrate multi-view constraints into single-view 2D convolutional networks, achieving state-of-the-art depth estimation accuracy while being up to 48× faster than prior methods. The approach uses epipolar-line attention to fuse features across views with minimal computational overhead.

ABSTRACT

Deep learning has made significant impacts on multi-view stereo systems. State-of-the-art approaches typically involve building a cost volume, followed by multiple 3D convolution operations to recover the input image's pixel-wise depth. While such end-to-end learning of plane-sweeping stereo advances public benchmarks' accuracy, they are typically very slow to compute. We present \ouralg, a highly efficient multi-view stereo algorithm that seamlessly integrates multi-view constraints into single-view networks via an attention mechanism. Since \ouralg only builds on 2D convolutions, it is at least $2 imes$ faster than all the notable counterparts. Moreover, our algorithm produces precise depth estimations and 3D reconstructions, achieving state-of-the-art results on challenging benchmarks ScanNet, SUN3D, RGBD, and the classical DTU dataset. our algorithm also out-performs all other algorithms in the setting of inexact camera poses. Our code is released at \url{https://github.com/zhenpeiyang/MVS2D}

Motivation & Objective

• To address the high computational cost of state-of-the-art 3D convolution-based multi-view stereo (MVS) methods while maintaining or improving accuracy.
• To seamlessly integrate multi-view geometric constraints into single-view 2D CNN architectures without relying on 3D cost volumes or global scene representations.
• To improve robustness to inaccurate camera poses by enabling multi-scale feature aggregation in the attention mechanism.
• To achieve high inference speed without sacrificing depth estimation quality on challenging benchmarks.

Proposed method

• Introduces an Epipolar Attention Module that aggregates features along epipolar lines from multiple reference images to a query pixel in the reference image.
• Uses 2D convolutions exclusively throughout the network, avoiding expensive 3D convolutions and enabling faster inference.
• Employs learnable depth encoding to represent depth hypotheses, jointly optimized with the network for improved feature representation.
• Applies multi-scale attention modules in the encoder-decoder architecture to improve robustness when camera poses are inaccurate.
• Integrates the attention module into a standard 2D U-Net backbone, allowing easy deployment in existing single-view depth estimation frameworks.
• Leverages early fusion of single-view and multi-view features to enrich intermediate representations with 3D geometric cues.

Experimental results

Research questions

• RQ1 Can multi-view stereo be achieved with only 2D convolutions while maintaining state-of-the-art accuracy and significantly improving inference speed?
• RQ2 How effective is an attention mechanism that operates along epipolar lines in fusing multi-view features without explicit 3D cost volume construction?
• RQ3 To what extent does multi-scale attention improve robustness under noisy or inaccurate camera poses?
• RQ4 Can end-to-end learned depth encoding outperform fixed encodings like cosine or linear in MVS tasks?

Key findings

• MVS2D achieves state-of-the-art performance on ScanNet, SUN3D, RGBD, and DTU benchmarks, with an AbsRel of 0.059 on ScanNet and 0.059 on DTU under noisy poses.
• The model runs at 36.4 FPS on 640×480 resolution on DTU, which is 2.2× faster than PatchmatchNet and 4.7× faster than MVSNet.
• On the DTU validation set, MVS2D achieves RMSE of 14.769 mm and 71.8% of errors fall below 1.0 mm threshold, outperforming PatchmatchNet (32.348 mm RMSE).
• The ablation study shows that learned depth encoding improves AbsRel from 0.139 (uniform) to 0.059, demonstrating its critical role in performance.
• In the noisy pose setting, MVS2D with multi-scale attention (Ours-robust) achieves 0.059 AbsRel, outperforming MVSNet (0.094) and DPSNet (0.094), showing strong robustness.
• The method is 48× faster than NAS, 39× faster than DPSNet, and 10× faster than MVSNet, while maintaining SOTA accuracy.