[Paper Review] MVSFormer: Multi-View Stereo by Learning Robust Image Features and Temperature-based Depth

Chenjie Cao, Xinlin Ren+1 more|arXiv (Cornell University)|Aug 4, 2022

Advanced Vision and Imaging37 citations

TL;DR

MVSFormer introduces pre-trained Vision Transformers to enhance MVS feature learning and unifies regression and classification depth with a temperature-based depth inference, achieving state-of-the-art results on DTU and Tanks-and-Temples datasets.

ABSTRACT

Feature representation learning is the key recipe for learning-based Multi-View Stereo (MVS). As the common feature extractor of learning-based MVS, vanilla Feature Pyramid Networks (FPNs) suffer from discouraged feature representations for reflection and texture-less areas, which limits the generalization of MVS. Even FPNs worked with pre-trained Convolutional Neural Networks (CNNs) fail to tackle these issues. On the other hand, Vision Transformers (ViTs) have achieved prominent success in many 2D vision tasks. Thus we ask whether ViTs can facilitate feature learning in MVS? In this paper, we propose a pre-trained ViT enhanced MVS network called MVSFormer, which can learn more reliable feature representations benefited by informative priors from ViT. The finetuned MVSFormer with hierarchical ViTs of efficient attention mechanisms can achieve prominent improvement based on FPNs. Besides, the alternative MVSFormer with frozen ViT weights is further proposed. This largely alleviates the training cost with competitive performance strengthened by the attention map from the self-distillation pre-training. MVSFormer can be generalized to various input resolutions with efficient multi-scale training strengthened by gradient accumulation. Moreover, we discuss the merits and drawbacks of classification and regression-based MVS methods, and further propose to unify them with a temperature-based strategy. MVSFormer achieves state-of-the-art performance on the DTU dataset. Particularly, MVSFormer ranks as Top-1 on both intermediate and advanced sets of the highly competitive Tanks-and-Temples leaderboard.

Motivation & Objective

• Motivate robust feature learning for MVS to handle reflections and texture-less regions.
• Explore using pre-trained Vision Transformers (ViTs) to augment CNN-based feature extraction in MVS.
• Develop a multi-scale training strategy to adapt ViT-based features to high-resolution MVS inputs.
• Propose a temperature-based depth prediction to unify regression and classification-based depth estimation.

Proposed method

• Integrates pre-trained ViTs (Twins or DINO) with a CNN-based FPN feature extractor to obtain complementary multi-scale features for MVS.
• Uses a simple feature fusion by adding ViT-derived features to the highest level of the FPN encoder.
• Implements an efficient multi-scale training strategy with gradient accumulation to enable ViT training on varying resolutions.
• Constructs multi-stage cost volumes with view-weighted feature correlations and visibility weights, followed by 3D U-Net regularization.
• Introduces a temperature-based inference for depth by softmax scaling of the cost volume with a fixed temperature schedule to unify classification and regression depths.
• Provides an alternative MVSFormer-P variant with a frozen DINO backbone and GLU-based fusion to reduce training cost.

Experimental results

Research questions

• RQ1 Can pre-trained ViTs improve MVS feature representations and generalization to reflections and texture-less regions?
• RQ2 Does a multi-scale ViT-based training strategy enable robust high-resolution MVS performance?
• RQ3 Can a temperature-based depth inference unify classification and regression approaches to improve depth accuracy and point clouds?
• RQ4 How does the ViT-enhanced MVSFormer compare to state-of-the-art methods on standard MVS benchmarks like DTU and Tanks-and-Temples?

Key findings

• MVSFormer with Twins-small significantly reduces reconstruction error on DTU compared to CNN pre-trained backbones (e.g., from 0.312 to 0.289 overall error in Tab. 1).
• MVSFormer ranks Top-1 on both intermediate and advanced sets of Tanks-and-Temples, achieving mean F-scores of 66.37 (intermediate) and 40.87 (advanced).
• Classification-based confidence maps provide robustness to outliers and reflections, and temperature-based depth (D_tmp) during inference smooths predictions and improves point clouds.
• Temperature-based depth inference unifies regression and classification depths, with fixed schedule {t^1,t^2,t^3,t^4} = {5, 2.5, 1.5, 1} improving depth predictions over purely REG or CLA.
• MVSFormer-P (frozen DINO) achieves competitive results with reduced training cost, still delivering strong performance.