[论文解读] 3D Former: Monocular Scene Reconstruction with 3D SDF Transformers

Weihao Yuan, Xiaodong Gu+3 more|arXiv (Cornell University)|Jan 31, 2023

Advanced Vision and Imaging被引用 4

一句话总结

本文提出3D Former，一种新颖的3D SDF变换器网络，通过使用稀疏窗口多头注意力机制替代3D CNN，提升了单目场景重建中的3D特征聚合效果。通过引入膨胀注意力机制以保持几何结构，并采用全局模块以建模长距离上下文，该方法在ScanNet数据集上实现了最先进性能，将网格误差在准确度上降低了41.8%，在完整度上降低了25.3%。

ABSTRACT

Monocular scene reconstruction from posed images is challenging due to the complexity of a large environment. Recent volumetric methods learn to directly predict the TSDF volume and have demonstrated promising results in this task. However, most methods focus on how to extract and fuse the 2D features to a 3D feature volume, but none of them improve the way how the 3D volume is aggregated. In this work, we propose an SDF transformer network, which replaces the role of 3D CNN for better 3D feature aggregation. To reduce the explosive computation complexity of the 3D multi-head attention, we propose a sparse window attention module, where the attention is only calculated between the non-empty voxels within a local window. Then a top-down-bottom-up 3D attention network is built for 3D feature aggregation, where a dilate-attention structure is proposed to prevent geometry degeneration, and two global modules are employed to equip with global receptive fields. The experiments on multiple datasets show that this 3D transformer network generates a more accurate and complete reconstruction, which outperforms previous methods by a large margin. Remarkably, the mesh accuracy is improved by 41.8%, and the mesh completeness is improved by 25.3% on the ScanNet dataset. Project page: https://weihaosky.github.io/sdfformer.

研究动机与目标

• 解决在大型复杂环境中，从带位姿的RGB图像进行精确且完整的单目3D场景重建的挑战。
• 通过用更高效的基于注意力的机制替代3D CNN，克服其在3D特征聚合方面的局限性。
• 降低3D多头注意力的计算复杂度，使其适用于大规模3D重建任务。
• 通过新型网络组件提升3D特征学习中的几何保真度与长距离依赖建模能力。
• 在ScanNet和TUM-RGBD等基准数据集上实现网格准确度与完整度的最先进性能。

提出的方法

• 提出一种稀疏窗口多头注意力模块，仅在局部3D窗口内的非空体素之间计算注意力，显著降低计算成本。
• 引入膨胀注意力机制，防止在自顶向下-自底向上3D变换器架构中下采样与上采样过程中的几何退化。
• 设计一种具有三级结构的自顶向下-自底向上3D变换器网络，用于对反投影的2D特征进行从粗到细的特征聚合。
• 在底层集成全局注意力模块与全局上下文编码模块，以增强长距离依赖关系与场景级理解能力。
• 采用截断有符号距离函数（TSDF）作为3D表示，并通过Marching Cubes算法预测最终表面。
• 端到端训练整个网络，从已知相机位姿的单目RGB图像序列回归TSDF体素。

Figure 1: The overview of the 3D reconstruction framework. The input images are extracted to features by a 2D backbone network, then the 2D features are back-projected and fused to 3D feature volumes, which are aggregated by our 3D SDF transformer and generate the reconstruction in a coarse-to-fine

实验结果

研究问题

• RQ1 在单目场景重建中，采用稀疏窗口注意力的3D变换器是否能比3D CNN实现更优的3D特征聚合？
• RQ2 如何使3D多头注意力在大规模3D重建任务中具备计算可行性？
• RQ3 在自顶向下-自底向上变换器架构中使用膨胀注意力是否能保持几何细节并防止形状退化？
• RQ4 在底层引入全局模块是否能提升稀疏3D特征体积中的长距离上下文建模能力？
• RQ5 将3D CNN替换为3D变换器在多大程度上能提升3D重建中网格的准确度与完整度？

主要发现

• 所提出的3D Former在ScanNet数据集上实现了41.8%的网格准确度误差相对降低，从0.055降至0.032。
• 在ScanNet数据集上，网格完整度提升了25.3%，从0.083降至0.062。
• 消融实验表明，全局模块与后置膨胀注意力模块的引入显著提升了完整度与细节恢复能力。
• 稀疏窗口注意力中的窗口大小为10时，在性能与计算成本之间实现了最佳平衡。
• 该方法在ICL-NUIM与TUM-RGBD数据集的多个指标上均优于先前的最先进方法，包括NeuralRecon与VoRTX。
• 定性结果表明，重建的网格在某些情况下比真实网格更密集、更准确、更完整，且对细粒度细节的保持更优。

Figure 2: (a) Illustration of the sparse window attention. For calculating the attention of the current voxel (in orange), we first sparsify the volume using the occupancy prediction from the coarser level, and then search the occupied voxels (in dark blue) within a small window. The attention is he