[Paper Review] MoGDE: Boosting Mobile Monocular 3D Object Detection with Ground Depth Estimation

Yunsong Zhou, Quan Liu+4 more|arXiv (Cornell University)|Mar 23, 2023

Advanced Neural Network Applications4 citations

TL;DR

MoGDE proposes a novel monocular 3D object detection framework that enhances accuracy and robustness in mobile settings by estimating ground depth using camera pose and perspective geometry, then fusing this depth information via a transformer-based RGB-D feature fusion network. It achieves state-of-the-art performance on the KITTI 3D benchmark, ranking first with significant gains, especially for far objects.

ABSTRACT

Monocular 3D object detection (Mono3D) in mobile settings (e.g., on a vehicle, a drone, or a robot) is an important yet challenging task. Due to the near-far disparity phenomenon of monocular vision and the ever-changing camera pose, it is hard to acquire high detection accuracy, especially for far objects. Inspired by the insight that the depth of an object can be well determined according to the depth of the ground where it stands, in this paper, we propose a novel Mono3D framework, called MoGDE, which constantly estimates the corresponding ground depth of an image and then utilizes the estimated ground depth information to guide Mono3D. To this end, we utilize a pose detection network to estimate the pose of the camera and then construct a feature map portraying pixel-level ground depth according to the 3D-to-2D perspective geometry. Moreover, to improve Mono3D with the estimated ground depth, we design an RGB-D feature fusion network based on the transformer structure, where the long-range self-attention mechanism is utilized to effectively identify ground-contacting points and pin the corresponding ground depth to the image feature map. We conduct extensive experiments on the real-world KITTI dataset. The results demonstrate that MoGDE can effectively improve the Mono3D accuracy and robustness for both near and far objects. MoGDE yields the best performance compared with the state-of-the-art methods by a large margin and is ranked number one on the KITTI 3D benchmark.

Motivation & Objective

• To address the challenge of low detection accuracy for far objects in monocular 3D object detection due to near-far disparity in monocular vision.
• To improve robustness of Mono3D detectors under dynamic camera pose variations common in mobile applications such as autonomous vehicles and drones.
• To leverage ground depth as prior knowledge to guide 3D object detection by estimating pixel-level depth from camera pose and perspective geometry.
• To design a flexible, plug-in module that enhances existing image-only Mono3D detectors with ground depth guidance.
• To achieve state-of-the-art performance on the KITTI 3D benchmark, particularly for hard and far-object detection.

Proposed method

• A pose detection network estimates camera pitch and roll from image features using vanishing point and horizon line detection to infer view direction.
• A pixel-level ground depth feature map is constructed using 3D-to-2D perspective projection, assuming a virtual scene with only sky and ground.
• A ground-aware transformer-based feature fusion network uses long-range self-attention to identify ground-contacting points and align depth features with image features.
• The RGB-D feature fusion module integrates estimated ground depth with image features to improve 3D bounding box prediction accuracy.
• The method is designed as a plug-in module, enabling integration with existing image-only Mono3D detectors without architectural overhaul.
• Camera pose variance is simulated in ablation studies to evaluate robustness, with performance degradation measured under varying pose conditions.

Experimental results

Research questions

• RQ1 Can ground depth estimation significantly improve the accuracy of monocular 3D object detection, especially for far objects?
• RQ2 How effective is the integration of estimated ground depth with image features via a transformer-based fusion mechanism?
• RQ3 To what extent does MoGDE improve robustness under dynamic camera pose variations in mobile scenarios?
• RQ4 Can the proposed ground depth estimation and fusion components be effectively plugged into existing image-only Mono3D detectors to boost performance?
• RQ5 Does the use of perspective geometry and pose estimation lead to more reliable depth priors in real-world mobile environments?

Key findings

• MoGDE achieves the highest performance on the KITTI 3D benchmark, ranking first among all state-of-the-art methods.
• The method improves AP3D by +5.32, +5.77, and +5.97 points on the easy, moderate, and hard categories, respectively, when applied to M3D-RPN.
• When extended to Kinematic3D, MoGDE boosts AP3D by +1.83 (easy), +2.44 (moderate), and +2.30 (hard), with APBEV gains of +1.97, +3.08, and +2.86.
• The model shows minimal performance degradation under large camera pose variance, demonstrating strong robustness compared to baseline detectors.
• Visualization confirms that the transformer attention mechanism effectively focuses on ground-contacting points, linking depth cues to object features.
• The plug-in design enables consistent performance gains across multiple existing Mono3D detectors, confirming the method's flexibility and generalization.