[Paper Review] PLUMENet: Efficient 3D Object Detection from Stereo Images

Yan Wang, Bin Yang+3 more|arXiv (Cornell University)|Jan 17, 2021

Advanced Neural Network Applications4 citations

TL;DR

PLUMENet proposes a unified 3D feature volume, PLUME, constructed directly in 3D metric space from stereo images to jointly optimize depth estimation and 3D object detection, achieving state-of-the-art 66.3% AP on KITTI with 4× faster inference (6.6 Hz) than prior methods by eliminating representation mismatch and enabling full-resolution feature utilization.

ABSTRACT

3D object detection is a key component of many robotic applications such as self-driving vehicles. While many approaches rely on expensive 3D sensors such as LiDAR to produce accurate 3D estimates, methods that exploit stereo cameras have recently shown promising results at a lower cost. Existing approaches tackle this problem in two steps: first depth estimation from stereo images is performed to produce a pseudo LiDAR point cloud, which is then used as input to a 3D object detector. However, this approach is suboptimal due to the representation mismatch, as the two tasks are optimized in two different metric spaces. In this paper we propose a model that unifies these two tasks and performs them in the same metric space. Specifically, we directly construct a pseudo LiDAR feature volume (PLUME) in 3D space, which is then used to solve both depth estimation and object detection tasks. Our approach achieves state-of-the-art performance with much faster inference times when compared to existing methods on the challenging KITTI benchmark.

Motivation & Objective

• Address the representation mismatch between image-space depth estimation and 3D-space object detection in stereo-based 3D detection.
• Reduce computational and memory costs associated with 3D cost volume construction in image space.
• Improve long-range detection performance by mitigating perspective distortion effects from image-space processing.
• Enable end-to-end joint optimization of depth and detection using a unified 3D feature volume in metric space.
• Achieve state-of-the-art accuracy with significantly faster inference for real-time autonomous driving applications.

Proposed method

• Construct a pseudo-LiDAR feature volume (PLUME) directly in 3D metric space from stereo image features, bypassing image-space cost volume construction.
• Use full-resolution stereo image features to build PLUME, avoiding downsampling and preserving spatial detail.
• Perform both depth estimation and 3D object detection within the same 3D/BEV space, enabling shared feature learning and joint optimization.
• Apply 3D and 2D convolutions in a 3D-BEV network architecture to extract discriminative features from PLUME for both tasks.
• Integrate image features via bilinear warping and fusion with 3D features to enhance detection performance.
• Use 3D occupancy prediction as supervision for depth estimation, reducing bias from perspective distortion in image space.

Experimental results

Research questions

• RQ1 Can a unified 3D feature volume representation in metric space improve both depth estimation and 3D detection performance compared to sequential image-to-3D pipelines?
• RQ2 Does constructing the feature volume directly in 3D space reduce computational cost and memory usage compared to image-space cost volume methods?
• RQ3 Can joint optimization in 3D space mitigate the perspective distortion bias that affects long-range depth estimation in image-based methods?
• RQ4 To what extent does using full-resolution image features in 3D space improve detection accuracy and inference speed?
• RQ5 How does the proposed PLUME representation compare to existing feature volume formats (e.g., disparity FV, depth FV, pseudo-LiDAR FV) in terms of accuracy and efficiency?

Key findings

• PLUMENet-Middle achieves 66.3% average precision (AP) in bird’s eye view at the moderate level on the KITTI test set, surpassing all prior methods without external training data.
• The model runs at 6.6 Hz (150 ms per frame), representing a 4× speedup over comparable high-accuracy baselines like CDN and DSGN.
• The large-scale PLUMENet-Large model significantly outperforms state-of-the-art methods, demonstrating the scalability of the PLUME approach.
• Ablation studies show that full-resolution image features yield better performance than downsampled features, with AP dropping as resolution decreases.
• Image feature fusion improves detection by 0.4 AP points, confirming the value of incorporating high-resolution image cues.
• Qualitative results show that PLUMENet-Large produces high-quality, precise occupancy predictions with correct object shapes and minimal false positives.