[论文解读] MVSFormer: Multi-View Stereo by Learning Robust Image Features and Temperature-based Depth

Chenjie Cao, Xinlin Ren+1 more|arXiv (Cornell University)|Aug 4, 2022

Advanced Vision and Imaging被引用 37

一句话总结

MVSFormer 引入预训练的 Vision Transformers 以增强 MVS 特征学习，并通过基于温度的深度推断统一回归与分类深度，在 DTU 与 Tanks-and-Temples 数据集上达到最先进的结果。

ABSTRACT

Feature representation learning is the key recipe for learning-based Multi-View Stereo (MVS). As the common feature extractor of learning-based MVS, vanilla Feature Pyramid Networks (FPNs) suffer from discouraged feature representations for reflection and texture-less areas, which limits the generalization of MVS. Even FPNs worked with pre-trained Convolutional Neural Networks (CNNs) fail to tackle these issues. On the other hand, Vision Transformers (ViTs) have achieved prominent success in many 2D vision tasks. Thus we ask whether ViTs can facilitate feature learning in MVS? In this paper, we propose a pre-trained ViT enhanced MVS network called MVSFormer, which can learn more reliable feature representations benefited by informative priors from ViT. The finetuned MVSFormer with hierarchical ViTs of efficient attention mechanisms can achieve prominent improvement based on FPNs. Besides, the alternative MVSFormer with frozen ViT weights is further proposed. This largely alleviates the training cost with competitive performance strengthened by the attention map from the self-distillation pre-training. MVSFormer can be generalized to various input resolutions with efficient multi-scale training strengthened by gradient accumulation. Moreover, we discuss the merits and drawbacks of classification and regression-based MVS methods, and further propose to unify them with a temperature-based strategy. MVSFormer achieves state-of-the-art performance on the DTU dataset. Particularly, MVSFormer ranks as Top-1 on both intermediate and advanced sets of the highly competitive Tanks-and-Temples leaderboard.

研究动机与目标

• 推动稳健的 MVS 特征学习以应对反射和纹理缺失区域。
• 探索使用预训练的 Vision Transformers (ViTs) 来增强基于 CNN 的 MVS 特征提取。
• Develop a multi-scale training strategy to adapt ViT-based features to high-resolution MVS inputs.
• 提出一种基于温度的深度预测以统一回归和基于分类的深度估计。

提出的方法

• 将预训练的 ViTs (Twins 或 DINO) 与基于 CNN 的 FPN 特征提取器集成，以获得用于 MVS 的互补多尺度特征。
• 通过将 ViT 派生的特征相加到 FPN 编码器的最高层来实现简单的特征融合。
• 实现一种高效的多尺度训练策略，结合梯度累积以在不同分辨率下实现 ViT 的训练。
• 构建带有视角加权特征相关性和可见性权重的多阶段代价体，随后进行 3D U-Net 正则化。
• 通过对代价体进行定温度调度的 softmax 缩放来实现基于温度的深度推断，以统一分类和回归深度。
• 提供一个替代的 MVSFormer-P 变体（冻结 DINO 主干并采用 GLU 融合）以降低训练成本。

实验结果

研究问题

• RQ1 预训练的 ViTs 是否能改善 MVS 的特征表示并提升对反射和纹理缺失区域的泛化能力？
• RQ2 多尺度的 ViT 基训练策略是否能实现高分辨率 MVS 的鲁棒性？
• RQ3 基于温度的深度推断是否能统一回归与分类方法，从而提高深度精度和点云质量？
• RQ4 ViT 增强的 MVSFormer 相较于 DTU 和 Tanks-and-Temples 等标准 MVS 基准在现有方法中的表现如何？

主要发现

• MVSFormer 采用 Twins-small 相较于基于 CNN 的预训练骨干在 DTU 上显著降低重建误差（如表 1 由 0.312 降至 0.289 的总体误差）。
• MVSFormer 在 Tanks-and-Temples 的中级集和高级集均排名第 1，平均 F-score 分别为 66.37（中级）和 40.87（高级）。
• 基于分类的置信度图对离群点和反射具有鲁棒性，推断阶段的温度深度估计（D_tmp）可平滑预测并改善点云。
• 基于温度的深度推断统一了回归与分类深度，固定调度 {t^1,t^2,t^3,t^4} = {5, 2.5, 1.5, 1} 相比纯 REG 或 CLA 提高了深度预测。
• MVSFormer-P（冻结 DINO）在降低训练成本的同时仍能取得具有竞争力的结果。