[Paper Review] Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud

Xinshuo Weng, Kris Kitani|arXiv (Cornell University)|Mar 23, 2019

Advanced Neural Network Applications53 references35 citations

TL;DR

This paper converts a single image into a pseudo-LiDAR point cloud via monocular depth estimation and uses LiDAR-based Frustum PointNets with bounding box consistency to achieve state-of-the-art monocular 3D detection on KITTI.

ABSTRACT

Monocular 3D scene understanding tasks, such as object size estimation, heading angle estimation and 3D localization, is challenging. Successful modern day methods for 3D scene understanding require the use of a 3D sensor. On the other hand, single image based methods have significantly worse performance. In this work, we aim at bridging the performance gap between 3D sensing and 2D sensing for 3D object detection by enhancing LiDAR-based algorithms to work with single image input. Specifically, we perform monocular depth estimation and lift the input image to a point cloud representation, which we call pseudo-LiDAR point cloud. Then we can train a LiDAR-based 3D detection network with our pseudo-LiDAR end-to-end. Following the pipeline of two-stage 3D detection algorithms, we detect 2D object proposals in the input image and extract a point cloud frustum from the pseudo-LiDAR for each proposal. Then an oriented 3D bounding box is detected for each frustum. To handle the large amount of noise in the pseudo-LiDAR, we propose two innovations: (1) use a 2D-3D bounding box consistency constraint, adjusting the predicted 3D bounding box to have a high overlap with its corresponding 2D proposal after projecting onto the image; (2) use the instance mask instead of the bounding box as the representation of 2D proposals, in order to reduce the number of points not belonging to the object in the point cloud frustum. Through our evaluation on the KITTI benchmark, we achieve the top-ranked performance on both bird's eye view and 3D object detection among all monocular methods, effectively quadrupling the performance over previous state-of-the-art. Our code is available at https://github.com/xinshuoweng/Mono3D_PLiDAR.

Motivation & Objective

• Bridge the performance gap between 3D sensing and 2D sensing by enabling LiDAR-based 3D detectors to work with single images.
• Generate a pseudo-LiDAR point cloud from monocular depth estimation and image lifting.
• Mitigate pseudo-LiDAR noise via 2D-3D bounding box consistency and instance mask proposals.
• Demonstrate state-of-the-art monocular 3D detection performance on KITTI BEV and 3D metrics.
• Provide ablation evidence for the key components: BBCL, BBCO, instance masks, and density effects.

Proposed method

• Lift every pixel of the input image to 3D using monocular depth estimation to create a pseudo-LiDAR point cloud.
• Detect 2D instance masks (instead of bounding boxes) and extract point cloud frustums from the pseudo-LiDAR for each proposal.
• Apply a Frustum PointNet-based amodal 3D object detector to predict 3D bounding box parameters (x,y,z,h,w,l,θ) for each frustum.
• Impose a 2D-3D bounding box consistency constraint during training via a Bounding Box Consistency Loss (BBCL).
• During testing, refine predictions with Bounding Box Consistency Optimization (BBCO) to maximize 2D IoU between projected 3D boxes and 2D proposals.
• Optionally remove the TNet component to study its impact on performance.

Experimental results

Research questions

• RQ1 Can monocular depth estimation be leveraged to produce a usable pseudo-LiDAR representation for 3D object detection?
• RQ2 Do 2D instance masks and a 2D-3D bounding box consistency constraint improve monocular 3D detection under pseudo-LiDAR?
• RQ3 How does pseudo-LiDAR density and 3D box refinement affect BEV and 3D detection metrics on KITTI?
• RQ4 What is the benefit of post-processing with BBCO and BBCL in training and testing?
• RQ5 How close can monocular methods get to LiDAR-based 3D detection performance on standard benchmarks?

Key findings

• The method achieves top-ranked AP_BE V and AP_3D among monocular methods on KITTI, with substantial gains over prior art.
• At IoU 0.5 for moderate cars, AP_3D improves from 5.7% (previous state) to 21.0% with the proposed approach.
• At IoU 0.5, BEV and 3D metrics for Ours reach 72.1% and 68.4% (Easy), 53.1% and 48.3% (Moderate), 44.6% and 43.0% (Hard).
• BBCL and BBCO provide significant gains over baseline +PLiDAR across Easy/Moderate/Hard for 3D detection at IoU 0.5 and 0.7.
• Using instance masks for 2D proposals reduces tail artifacts in the pseudo-LiDAR frustums and improves 3D size estimation.
• Removing TNet yields mixed results and can slightly improve some IoU0.7 metrics but does not consistently outperform the full design.