[Paper Review] Monocular 3D Object Detection: An Extrinsic Parameter Free Approach

Yunsong Zhou, Yuan He+4 more|arXiv (Cornell University)|Jun 30, 2021

Advanced Neural Network Applications60 references4 citations

TL;DR

This paper proposes MonoEF, a monocular 3D object detection framework that eliminates sensitivity to camera extrinsic parameter perturbations by estimating camera pose via vanishing point and horizon detection from a single image. By using a learned feature transformation network guided by estimated extrinsic parameters, the method achieves state-of-the-art performance on KITTI 3D and nuScenes, significantly improving accuracy on uneven roads where existing methods fail due to pose-induced distortions.

ABSTRACT

Monocular 3D object detection is an important task in autonomous driving. It can be easily intractable where there exists ego-car pose change w.r.t. ground plane. This is common due to the slight fluctuation of road smoothness and slope. Due to the lack of insight in industrial application, existing methods on open datasets neglect the camera pose information, which inevitably results in the detector being susceptible to camera extrinsic parameters. The perturbation of objects is very popular in most autonomous driving cases for industrial products. To this end, we propose a novel method to capture camera pose to formulate the detector free from extrinsic perturbation. Specifically, the proposed framework predicts camera extrinsic parameters by detecting vanishing point and horizon change. A converter is designed to rectify perturbative features in the latent space. By doing so, our 3D detector works independent of the extrinsic parameter variations and produces accurate results in realistic cases, e.g., potholed and uneven roads, where almost all existing monocular detectors fail to handle. Experiments demonstrate our method yields the best performance compared with the other state-of-the-arts by a large margin on both KITTI 3D and nuScenes datasets.

Motivation & Objective

• Address the critical challenge of camera extrinsic parameter fluctuations in real-world autonomous driving scenarios, particularly on uneven or potholed roads.
• Overcome the limitation of existing monocular 3D detectors that assume fixed camera poses and fail under dynamic road-induced pose changes.
• Develop a method that enables 3D detection to be robust to extrinsic parameter variations without relying on LiDAR, radar, or multi-camera systems.
• Integrate camera pose estimation directly into the detection pipeline to implicitly correct feature distortions caused by road-induced camera tilt and rotation.
• Achieve state-of-the-art performance on standard benchmarks while maintaining real-time inference efficiency.

Proposed method

• Estimate camera extrinsic parameters (tilt and rotation relative to the ground plane) by detecting the vanishing point and horizon line from a single monocular image.
• Train a dedicated extrinsic regression network to predict camera pose from image features, using supervised learning on KITTI odometry sequences.
• Design a feature transformation network that uses the predicted extrinsic parameters as a seed to rectify latent features in the feature space.
• Apply the transformation network during inference to correct for perturbations in the feature maps before 3D box prediction.
• Integrate the extrinsic-aware module into a base detector (e.g., SMOKE) without modifying its backbone, enabling plug-and-play enhancement.
• Project the corrected 3D predictions back into the real-world coordinate system using the estimated camera geometry.

Experimental results

Research questions

• RQ1 Can monocular 3D detection be made robust to camera extrinsic parameter variations caused by road irregularities?
• RQ2 To what extent can vanishing point and horizon detection from a single image improve 3D detection accuracy under pose perturbations?
• RQ3 Does incorporating estimated camera pose into the feature space improve generalization on real-world uneven road conditions?
• RQ4 Can a learned feature transformation network effectively correct for extrinsic-induced distortions without requiring ground-truth extrinsic data during training?
• RQ5 How does the proposed method compare to SOTA monocular detectors in terms of accuracy and robustness on benchmarks with and without pose variation?

Key findings

• MonoEF achieves a 41.78% AP3D on the KITTI 3D benchmark, significantly outperforming the baseline SMOKE (34.98%) and SOTA methods like M3D-RPN (+E.F.: 41.36%) and Kinematic3D (+E.F.: 48.92%).
• On the KITTI odometry sequence 08, the model achieves comparable camera pose estimation accuracy to SOTA visual odometry methods, with angular errors of 0.87° for pitch and 0.92° for roll.
• The feature rectification module reduces ATE (Average Translation Error) and ASE (Average Scale Error) by a large margin on nuScenes, demonstrating improved 3D localization accuracy.
• The ablation study confirms that extrinsic parameter correction leads to substantial performance gains, especially on single-sequence training where initial pose inconsistency is high.
• MonoEF maintains low inference time, outperforming methods requiring additional operations like pseudo-LiDAR generation or pair-based refinement.
• When applied to other SOTA detectors (e.g., M3D-RPN, Kinematic3D), the extrinsic correction module consistently improves performance, proving its generalizability.