[Paper Review] MonoDTR: Monocular 3D Object Detection with Depth-Aware Transformer

Kuan‐Chih Huang, Tsung-Han Wu+2 more|arXiv (Cornell University)|Mar 21, 2022

Advanced Neural Network Applications10 citations

TL;DR

MonoDTR proposes an end-to-end monocular 3D object detection framework using a depth-aware transformer network that learns depth-aware features via auxiliary supervision without relying on off-the-shelf depth estimators. It introduces a lightweight Depth-Aware Feature Enhancement (DFE) module and a Depth-Aware Transformer (DTR) with a novel Depth Positional Encoding (DPE), achieving state-of-the-art performance on KITTI with real-time inference.

ABSTRACT

Monocular 3D object detection is an important yet challenging task in autonomous driving. Some existing methods leverage depth information from an off-the-shelf depth estimator to assist 3D detection, but suffer from the additional computational burden and achieve limited performance caused by inaccurate depth priors. To alleviate this, we propose MonoDTR, a novel end-to-end depth-aware transformer network for monocular 3D object detection. It mainly consists of two components: (1) the Depth-Aware Feature Enhancement (DFE) module that implicitly learns depth-aware features with auxiliary supervision without requiring extra computation, and (2) the Depth-Aware Transformer (DTR) module that globally integrates context- and depth-aware features. Moreover, different from conventional pixel-wise positional encodings, we introduce a novel depth positional encoding (DPE) to inject depth positional hints into transformers. Our proposed depth-aware modules can be easily plugged into existing image-only monocular 3D object detectors to improve the performance. Extensive experiments on the KITTI dataset demonstrate that our approach outperforms previous state-of-the-art monocular-based methods and achieves real-time detection. Code is available at https://github.com/kuanchihhuang/MonoDTR

Motivation & Objective

• Address the limitations of existing depth-assisted monocular 3D detection methods that suffer from inaccurate depth priors and high computational cost.
• Develop a lightweight, end-to-end framework that implicitly learns depth-aware features without requiring pre-trained depth estimators during inference.
• Enhance 3D detection performance by integrating context- and depth-aware features globally using a transformer-based architecture.
• Introduce a novel Depth Positional Encoding (DPE) to better represent depth-aware spatial relationships in 3D object detection.
• Ensure compatibility and performance gains when plugged into existing image-only monocular 3D detectors.

Proposed method

• Proposes a Depth-Aware Feature Enhancement (DFE) module that learns depth-aware features using auxiliary depth supervision during training, avoiding reliance on external depth estimators.
• Introduces a Depth-Aware Transformer (DTR) module that globally attends to and fuses context-aware and depth-aware features using a transformer encoder-decoder structure.
• Designs a novel Depth Positional Encoding (DPE) that injects depth-aware positional cues into the transformer, improving 3D localization over standard pixel-wise positional encodings.
• Uses depth-aware features as input queries in the transformer decoder to enhance 3D reasoning and object localization accuracy.
• Employs a multi-stage training strategy where depth supervision is applied only during training, enabling real-time inference without depth estimation overhead.
• Enables plug-and-play integration into existing image-only monocular 3D detectors by replacing their detection head with the proposed DFE and DTR modules.

Experimental results

Research questions

• RQ1 Can depth-aware features be learned effectively without relying on pre-trained depth estimators, thereby reducing computational cost and error propagation?
• RQ2 How does a transformer-based fusion mechanism compare to conventional convolutional fusion in integrating context and depth-aware features for 3D detection?
• RQ3 Does a depth-aware positional encoding (DPE) outperform standard positional encodings (e.g., sinusoidal, absolute) in 3D object detection tasks?
• RQ4 To what extent can the proposed depth-aware modules improve performance when integrated into existing image-only monocular 3D detectors?
• RQ5 Can the proposed framework achieve real-time inference while maintaining state-of-the-art accuracy on benchmark datasets like KITTI?

Key findings

• MonoDTR achieves state-of-the-art performance on the KITTI 3D detection benchmark, with 24.52% AP3D@IoU=0.7 on the hard set for the car category.
• The proposed DPE outperforms all baseline positional encodings, achieving 24.52% AP3D@IoU=0.7 (hard) compared to 24.34% for CPE and 23.85% for APE.
• When plugged into M3D-RPN, GAC, and MonoDLE, the proposed modules improve AP3D@IoU=0.7 by up to +6.43% on the hard set, demonstrating strong generalization and compatibility.
• The model achieves real-time inference speed, making it practical for real-world autonomous driving applications.
• Qualitative results show that predictions from MonoDTR are significantly closer to ground truth in both image and bird’s-eye view, especially for distant or occluded objects.
• Ablation studies confirm that the DFE and DTR modules contribute substantially to performance gains, with DPE providing a consistent improvement across all settings.