[Paper Review] DepthFormer: Exploiting Long-Range Correlation and Local Information for Accurate Monocular Depth Estimation

Zhenyu Li, Zehui Chen+2 more|arXiv (Cornell University)|Mar 27, 2022

Advanced Vision and Imaging5 citations

TL;DR

DepthFormer proposes a hybrid Transformer-CNN encoder for monocular depth estimation, leveraging long-range correlation via self-attention and local detail preservation through a convolutional branch, with a novel Hierarchical Aggregation and Heterogeneous Interaction (HAHI) module to enhance feature fusion. It achieves state-of-the-art performance on KITTI, NYU, and SUN RGB-D benchmarks, notably setting a new SOTA on KITTI with improved accuracy and efficiency.

ABSTRACT

This paper aims to address the problem of supervised monocular depth estimation. We start with a meticulous pilot study to demonstrate that the long-range correlation is essential for accurate depth estimation. Therefore, we propose to leverage the Transformer to model this global context with an effective attention mechanism. We also adopt an additional convolution branch to preserve the local information as the Transformer lacks the spatial inductive bias in modeling such contents. However, independent branches lead to a shortage of connections between features. To bridge this gap, we design a hierarchical aggregation and heterogeneous interaction module to enhance the Transformer features via element-wise interaction and model the affinity between the Transformer and the CNN features in a set-to-set translation manner. Due to the unbearable memory cost caused by global attention on high-resolution feature maps, we introduce the deformable scheme to reduce the complexity. Extensive experiments on the KITTI, NYU, and SUN RGB-D datasets demonstrate that our proposed model, termed DepthFormer, surpasses state-of-the-art monocular depth estimation methods with prominent margins. Notably, it achieves the most competitive result on the highly competitive KITTI depth estimation benchmark. Our codes and models are available at https://github.com/zhyever/Monocular-Depth-Estimation-Toolbox.

Motivation & Objective

• To address the limitations of CNNs in modeling long-range correlations and Transformers in lacking spatial inductive bias for local details in monocular depth estimation.
• To improve depth estimation accuracy by jointly leveraging the strengths of Transformers (global context) and CNNs (local structure).
• To overcome feature fusion bottlenecks from late fusion of independent branches via a dedicated interaction mechanism.
• To reduce memory cost of global self-attention on high-resolution features using a deformable attention scheme.
• To establish a new state-of-the-art on major depth estimation benchmarks, especially KITTI.

Proposed method

• The model uses a dual-branch encoder: a Vision Transformer branch to model long-range dependencies and a CNN branch to preserve local spatial details.
• A Hierarchical Aggregation and Heterogeneous Interaction (HAHI) module is introduced to enhance features via multi-level self-attention and cross-attention between Transformer and CNN features.
• The HAHI module employs a deformable self-attention (DSA) mechanism to reduce computational cost by attending to a learnable subset of key points instead of all pixels.
• A deformable cross-attention (DCA) module models affinity between heterogeneous features in a set-to-set translation manner, enabling structured feature interaction.
• The framework uses standard decoder heads for depth prediction and is trained end-to-end with supervised loss.
• The deformable attention mechanism is applied across all hierarchical stages to maintain high-resolution feature representation with reduced memory footprint.

Experimental results

Research questions

• RQ1 Can a hybrid Transformer-CNN encoder outperform standard CNN or pure Transformer encoders in monocular depth estimation?
• RQ2 How effectively can a dedicated feature interaction module like HAHI improve fusion between long-range and local features?
• RQ3 Does the use of deformable attention reduce memory cost while preserving performance on high-resolution feature maps?
• RQ4 Can the proposed method achieve state-of-the-art performance on standard benchmarks like KITTI, NYU, and SUN RGB-D?
• RQ5 What is the relative contribution of long-range modeling versus local detail preservation in depth estimation accuracy?

Key findings

• DepthFormer achieves the new state-of-the-art on the highly competitive KITTI depth estimation benchmark, demonstrating superior accuracy.
• The model improves RMS error by 4.9% when using multi-level deformable self-attention (DSA) compared to single-level attention.
• The addition of deformable cross-attention (DCA) yields a 2.2% improvement in RMS error, confirming its effectiveness in heterogeneous feature interaction.
• The HAHI module significantly enhances performance, especially for near-object depth estimation, where local consistency is critical.
• DepthFormer maintains high inference speed, outperforming prior SOTA methods in both accuracy (δ₁) and speed on the KITTI validation set.
• The ablation study confirms that the combination of multi-level DSA and DCA is essential for optimal performance, as removing either degrades results substantially.