[Paper Review] MonoJSG: Joint Semantic and Geometric Cost Volume for Monocular 3D Object Detection

Qing Lian, Peiliang Li+1 more|arXiv (Cornell University)|Mar 16, 2022

Advanced Vision and Imaging4 citations

TL;DR

MonoJSG proposes a joint semantic and geometric cost volume for monocular 3D object detection that refines depth predictions using pixel-level visual cues. By projecting normalized 3D object coordinates into image space and modeling both geometric and semantic errors in an adaptive 4D cost volume, the method achieves state-of-the-art performance on KITTI and Waymo benchmarks with real-time inference.

ABSTRACT

Due to the inherent ill-posed nature of 2D-3D projection, monocular 3D object detection lacks accurate depth recovery ability. Although the deep neural network (DNN) enables monocular depth-sensing from high-level learned features, the pixel-level cues are usually omitted due to the deep convolution mechanism. To benefit from both the powerful feature representation in DNN and pixel-level geometric constraints, we reformulate the monocular object depth estimation as a progressive refinement problem and propose a joint semantic and geometric cost volume to model the depth error. Specifically, we first leverage neural networks to learn the object position, dimension, and dense normalized 3D object coordinates. Based on the object depth, the dense coordinates patch together with the corresponding object features is reprojected to the image space to build a cost volume in a joint semantic and geometric error manner. The final depth is obtained by feeding the cost volume to a refinement network, where the distribution of semantic and geometric error is regularized by direct depth supervision. Through effectively mitigating depth error by the refinement framework, we achieve state-of-the-art results on both the KITTI and Waymo datasets.

Motivation & Objective

• Address the ill-posed nature of monocular 3D object detection due to ambiguous 2D-3D depth recovery.
• Overcome limitations of existing methods that rely on sparse 2D cues or indirect depth supervision.
• Leverage dense pixel-level visual cues (geometric and semantic) to refine depth predictions more accurately.
• Integrate end-to-end depth supervision to improve feature discriminability for refinement.
• Achieve state-of-the-art performance on KITTI and Waymo datasets with real-time inference capability.

Proposed method

• Estimate 3D object coordinates, dimensions, and yaw angles using a deep neural network to generate normalized 3D object features.
• Reproject normalized 3D coordinates into the image space using initial depth estimates to form pixel-level geometric constraints.
• Measure geometric error as the difference between predicted 2D pixel locations and projected 3D coordinates.
• Compute semantic error by comparing features extracted at the original 2D location and the projected 3D coordinate location.
• Construct an adaptive 4D cost volume that models the joint distribution of geometric and semantic errors around candidate depths.
• Use a refinement network with depth supervision to optimize the cost volume and predict final depth, with adaptive sampling based on predicted depth uncertainty.

Experimental results

Research questions

• RQ1 Can dense pixel-level visual cues improve depth estimation accuracy in monocular 3D object detection compared to sparse 2D keypoint constraints?
• RQ2 How does combining semantic and geometric errors in a cost volume enhance depth refinement compared to using either cue alone?
• RQ3 To what extent does end-to-end depth supervision improve the discriminative quality of features used in refinement?
• RQ4 Can an adaptive cost volume that varies per proposal improve generalization and accuracy across diverse depth and object scales?
• RQ5 Does the proposed method achieve state-of-the-art performance on standard benchmarks like KITTI and Waymo while maintaining real-time inference?

Key findings

• MonoJSG achieves 26.4% mAP on the KITTI test set (Easy) and 18.3% on the Moderate set, setting a new SOTA.
• On the Waymo dataset, MonoJSG achieves 26.4% mAP (Easy), 18.3% (Moderate), and 15.4% (Hard), surpassing prior methods.
• The ablation study shows that using both geometric and semantic features in the refinement module improves mAP by 2.1% over geometric-only features.
• Adaptive sampling in the cost volume outperforms uniform sampling, with a 1.1% mAP gain on the KITTI Moderate set when using 32 samples.
• The method reduces depth error significantly, as shown by more convex loss landscapes and better alignment with ground truth in qualitative results.
• The model shows robustness to occlusion and textureless regions by leveraging learned semantic representations over raw photometric features.