[Paper Review] SparseFormer: Attention-based Depth Completion Network

Frederik Warburg, Michaël Ramamonjisoa+1 more|arXiv (Cornell University)|Jun 9, 2022

Advanced Vision and Imaging4 citations

TL;DR

SparseFormer proposes a transformer-based network for very sparse depth completion using 3D Structure-from-Motion (SfM) landmarks and RGB images. It employs a global attention mechanism to interpolate depth across non-uniform, sparse landmarks and a refinement module with cross-attention to filter outliers, achieving state-of-the-art performance on both indoor (NYU Depth V2) and large-scale outdoor (Mapillary Planet-Scale Depth) datasets at ultra-low point densities.

ABSTRACT

Most pipelines for Augmented and Virtual Reality estimate the ego-motion of the camera by creating a map of sparse 3D landmarks. In this paper, we tackle the problem of depth completion, that is, densifying this sparse 3D map using RGB images as guidance. This remains a challenging problem due to the low density, non-uniform and outlier-prone 3D landmarks produced by SfM and SLAM pipelines. We introduce a transformer block, SparseFormer, that fuses 3D landmarks with deep visual features to produce dense depth. The SparseFormer has a global receptive field, making the module especially effective for depth completion with low-density and non-uniform landmarks. To address the issue of depth outliers among the 3D landmarks, we introduce a trainable refinement module that filters outliers through attention between the sparse landmarks.

Motivation & Objective

• Address the challenge of depth completion when input 3D landmarks are extremely sparse, non-uniformly distributed, and prone to outliers—common in SfM pipelines.
• Design a method that generalizes across varying densities and spatial distributions of sparse 3D points.
• Overcome limitations of convolutional networks in handling very sparse data by leveraging the global receptive field of transformers.
• Improve depth completion accuracy by fusing visual features with sparse 3D landmarks through attention-based interpolation.
• Develop a trainable refinement module to detect and suppress depth outliers in SfM landmarks before interpolation.

Proposed method

• Introduce the SparseFormer module, which computes an attention volume between 2D-projected 3D SfM landmarks and visual features from a CNN decoder to enable global, similarity-based depth interpolation.
• Use a multi-head cross-attention mechanism between sparse 3D landmarks and image features to compute attention weights that guide depth propagation across the entire image.
• Integrate a refinement module that concatenates depth values with visual features at each SfM landmark and applies cross-attention to filter noisy or erroneous points.
• Insert four SparseFormer blocks into the decoder of Monodepth2, with multi-scale supervision and weighted loss to improve feature learning and depth accuracy.
• Train the model using Adam optimizer with learning rate decay, batch size 24, and 600k iterations on NYU Depth V2 and Mapillary Planet-Scale Depth datasets.
• Leverage the global context of self-attention to diffuse depth effectively even when only 300 points (0.1% density) are available per frame.

Experimental results

Research questions

• RQ1 Can a transformer-based module achieve superior depth completion performance on very sparse, non-uniformly distributed SfM landmarks compared to convolutional methods?
• RQ2 How does the global attention mechanism in SparseFormer handle extreme sparsity (e.g., <0.1% of pixels with depth) and non-uniform landmark distributions?
• RQ3 To what extent can a trainable refinement module based on cross-attention reduce the impact of depth outliers in SfM landmarks on the final prediction?
• RQ4 Does the attention-based interpolation mechanism generalize across diverse scenes and datasets with varying landmark densities and spatial distributions?
• RQ5 How does the proposed method compare to state-of-the-art baselines like ENet and Monodepth2 in terms of quantitative metrics and qualitative depth map quality?

Key findings

• On the NYU Depth V2 dataset, SparseFormer achieves a REL of 0.011, RMSE of 4.948, and δ1 of 0.989, outperforming ENet (REL: 0.019, RMSE: 5.469, δ1: 0.975) and Monodepth2 at low point densities.
• On the Mapillary Planet-Scale Depth dataset, SparseFormer significantly outperforms both Monodepth2 and ENet, demonstrating robustness on large-scale, real-world outdoor scenes with 0.1% landmark density.
• The model maintains high accuracy even with only 330 SfM landmarks per frame (0.1% density), showing strong generalization to ultra-sparse inputs.
• Qualitative results on MPSD show that SparseFormer produces coherent, detailed depth maps that align well with ground truth, even in textureless or occluded regions.
• Ablation studies confirm that the refinement module improves performance by filtering outliers, especially when landmark quality is low.
• The global attention mechanism enables effective depth diffusion across large image regions even with minimal input points, unlike local convolutional operations.