[論文レビュー] Improving Depth Gradient Continuity in Transformers: A Comparative Study on Monocular Depth Estimation with CNN

Jiawei Yao, Tong Wu+1 more|arXiv (Cornell University)|Aug 16, 2023

Advanced Vision and Imaging被引用数 45

ひとこと要約

Transformers はモノスクラ depth 推定におけるグローバルな文脈には優れるが、深度勾配の連続性には課題がある。本論文はプラグアンドプレイの Depth Gradient Refinement (DGR) モジュールと Optimal Transport Depth Loss (OTDL) を導入し、Transformer ベースの深度推定を強化、NYU-Depth-V2 と KITTI で最先端の結果を達成。

ABSTRACT

Monocular depth estimation is an ongoing challenge in computer vision. Recent progress with Transformer models has demonstrated notable advantages over conventional CNNs in this area. However, there's still a gap in understanding how these models prioritize different regions in 2D images and how these regions affect depth estimation performance. To explore the differences between Transformers and CNNs, we employ a sparse pixel approach to contrastively analyze the distinctions between the two. Our findings suggest that while Transformers excel in handling global context and intricate textures, they lag behind CNNs in preserving depth gradient continuity. To further enhance the performance of Transformer models in monocular depth estimation, we propose the Depth Gradient Refinement (DGR) module that refines depth estimation through high-order differentiation, feature fusion, and recalibration. Additionally, we leverage optimal transport theory, treating depth maps as spatial probability distributions, and employ the optimal transport distance as a loss function to optimize our model. Experimental results demonstrate that models integrated with the plug-and-play Depth Gradient Refinement (DGR) module and the proposed loss function enhance performance without increasing complexity and computational costs on both outdoor KITTI and indoor NYU-Depth-v2 datasets. This research not only offers fresh insights into the distinctions between Transformers and CNNs in depth estimation but also paves the way for novel depth estimation methodologies.

研究の動機と目的

• Compare Transformer and CNN performance in monocular depth estimation using visualization to identify focus regions and cues.
• Diagnose how Transformers differ from CNNs in handling depth gradients and boundaries.
• Propose and validate a plug-and-play Depth Gradient Refinement (DGR) module to improve gradient continuity in Transformer-based depth estimation.
• Introduce and evaluate an Optimal Transport Depth Loss (OTDL) to optimize depth distribution preservation during training.

提案手法

• Use sparse pixel masks to identify regions of interest for Transformer and CNN depth predictors.
• Develop and integrate Depth Gradient Refinement (DGR) that merges higher-order depth derivatives with feature recalibration after each Transformer encoder block.
• Define and apply Optimal Transport Depth Loss (OTDL) to compare predicted and ground-truth depth maps as normalized distributions with a quadratic cost Mij=|i−j|^2.
• Combine L_MSE with L_OTDL to form the final loss for training depth estimation models.
• Evaluate DGR and OT-based loss across multiple Transformer-based backbones (e.g., Adabins, DPT, TransDepth, PixelFormer, DepthFormer) on NYU-Depth-V2 and KITTI datasets.

実験結果

リサーチクエスチョン

• RQ1 What cues do Transformer-based monocular depth models rely on, and how do these cues differ from CNN-based cues?
• RQ2 Are Transformers more sensitive to image boundaries and global context yet less capable of maintaining depth gradient continuity?
• RQ3 Can the Depth Gradient Refinement (DGR) module improve depth gradient continuity without increasing model complexity?
• RQ4 Does the Optimal Transport Depth Loss (OTDL) complement standard MSE loss to improve depth distribution continuity and overall accuracy?

主な発見

• Transformers focus more on object boundaries and gradients, showing clearer depth cues at edges but may produce unnatural depth jumps in smooth regions.
• With sparse input regions, Transformers maintain higher depth estimation performance than CNNs at equivalent sparsity, and remain more robust as regions are masked.
• Incorporating DGR improves depth boundaries and gradient continuity across Transformer models, yielding state-of-the-art results on NYU-Depth-V2 and KITTI when paired with strong backbones (e.g., PixelFormer + DGR).
• On NYU-Depth-V2, PixelFormer + DGR achieves Abs Rel 0.086 and RMSE 0.310 (delta1 0.937); Adabins + DGR improves Abs Rel to 0.097 and RMSE to 0.347.
• On KITTI, DepthFormer + DGR achieves Abs Rel 0.050 and RMSE 2.124 (delta1 0.979), while PixelFormer + DGR achieves RMSE 2.041 (Abs Rel 0.049).
• Training with a combination of L_MSE and L_OTDL yields best performance across evaluated models on NYU-Depth-V2.