[Paper Review] 3D Former: Monocular Scene Reconstruction with 3D SDF Transformers

Weihao Yuan, Xiaodong Gu+3 more|arXiv (Cornell University)|Jan 31, 2023

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes 3D Former, a novel 3D SDF transformer network that replaces 3D CNNs with a sparse window multi-head attention mechanism for improved 3D feature aggregation in monocular scene reconstruction. By introducing dilate-attention for geometry preservation and global modules for long-range context, it achieves state-of-the-art performance, reducing mesh error by 41.8% in accuracy and 25.3% in completeness on ScanNet.

ABSTRACT

Monocular scene reconstruction from posed images is challenging due to the complexity of a large environment. Recent volumetric methods learn to directly predict the TSDF volume and have demonstrated promising results in this task. However, most methods focus on how to extract and fuse the 2D features to a 3D feature volume, but none of them improve the way how the 3D volume is aggregated. In this work, we propose an SDF transformer network, which replaces the role of 3D CNN for better 3D feature aggregation. To reduce the explosive computation complexity of the 3D multi-head attention, we propose a sparse window attention module, where the attention is only calculated between the non-empty voxels within a local window. Then a top-down-bottom-up 3D attention network is built for 3D feature aggregation, where a dilate-attention structure is proposed to prevent geometry degeneration, and two global modules are employed to equip with global receptive fields. The experiments on multiple datasets show that this 3D transformer network generates a more accurate and complete reconstruction, which outperforms previous methods by a large margin. Remarkably, the mesh accuracy is improved by 41.8%, and the mesh completeness is improved by 25.3% on the ScanNet dataset. Project page: https://weihaosky.github.io/sdfformer.

Motivation & Objective

• Address the challenge of accurate and complete monocular 3D scene reconstruction from posed RGB images in large, complex environments.
• Overcome the limitations of 3D CNNs in 3D feature aggregation by replacing them with a more effective attention-based mechanism.
• Reduce the computational complexity of 3D multi-head attention to make it feasible for large-scale 3D reconstruction tasks.
• Improve geometry preservation and long-range dependency modeling in 3D feature learning through novel architectural components.
• Achieve state-of-the-art performance in mesh accuracy and completeness on benchmark datasets like ScanNet and TUM-RGBD.

Proposed method

• Propose a sparse window multi-head attention module that computes attention only between non-empty voxels within localized 3D windows, drastically reducing computation cost.
• Introduce a dilate-attention mechanism to prevent geometry degeneration during downsampling and upsampling in a top-down-bottom-up 3D transformer architecture.
• Design a top-down-bottom-up 3D transformer network with three levels for coarse-to-fine feature aggregation of back-projected 2D features.
• Integrate a global attention module and a global context encoding module at the bottom level to enhance long-range dependencies and scene-level understanding.
• Use truncated signed distance function (TSDF) as the 3D representation and predict the final surface via Marching Cubes.
• Train the entire network end-to-end to regress the TSDF volume from a sequence of monocular RGB images with known camera poses.

Figure 1: The overview of the 3D reconstruction framework. The input images are extracted to features by a 2D backbone network, then the 2D features are back-projected and fused to 3D feature volumes, which are aggregated by our 3D SDF transformer and generate the reconstruction in a coarse-to-fine

Experimental results

Research questions

• RQ1 Can a 3D transformer with sparse window attention achieve better 3D feature aggregation than 3D CNNs in monocular scene reconstruction?
• RQ2 How can 3D multi-head attention be made computationally feasible for large-scale 3D reconstruction tasks?
• RQ3 Does the use of dilate-attention in a top-down-bottom-up transformer architecture preserve geometric details and prevent shape degeneration?
• RQ4 Can global modules at the bottom level improve long-range context modeling in sparse 3D feature volumes?
• RQ5 To what extent does replacing 3D CNNs with a 3D transformer improve mesh accuracy and completeness in 3D reconstruction?

Key findings

• The proposed 3D Former achieves a 41.8% relative reduction in mesh accuracy error on the ScanNet dataset, decreasing from 0.055 to 0.032.
• Mesh completeness improves by 25.3%, dropping from 0.083 to 0.062 on the ScanNet dataset.
• The ablation study confirms that the addition of the global module and post-dilate-attention block significantly improves completeness and detail recovery.
• The window size of 10 in the sparse window attention achieves optimal trade-off between performance and computational cost.
• The method outperforms prior state-of-the-art methods, including NeuralRecon and VoRTX, across multiple metrics on ICL-NUIM and TUM-RGBD datasets.
• Qualitative results show that the reconstructed meshes are denser, more accurate, and more complete than ground truth in some cases, with better preservation of fine details.

Figure 2: (a) Illustration of the sparse window attention. For calculating the attention of the current voxel (in orange), we first sparsify the volume using the occupancy prediction from the coarser level, and then search the occupied voxels (in dark blue) within a small window. The attention is he