[Paper Review] Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving

Yurong You, Yan Wang+6 more|arXiv (Cornell University)|Jun 14, 2019

Advanced Neural Network Applications48 references220 citations

TL;DR

The paper improves stereo-based depth with a depth-optimized stereo network (SDN) and a graph-based depth correction (GDC) using sparse LiDAR, boosting pseudo-LiDAR 3D detection and closing the gap to LiDAR-based methods on KITTI.

ABSTRACT

Detecting objects such as cars and pedestrians in 3D plays an indispensable role in autonomous driving. Existing approaches largely rely on expensive LiDAR sensors for accurate depth information. While recently pseudo-LiDAR has been introduced as a promising alternative, at a much lower cost based solely on stereo images, there is still a notable performance gap. In this paper we provide substantial advances to the pseudo-LiDAR framework through improvements in stereo depth estimation. Concretely, we adapt the stereo network architecture and loss function to be more aligned with accurate depth estimation of faraway objects --- currently the primary weakness of pseudo-LiDAR. Further, we explore the idea to leverage cheaper but extremely sparse LiDAR sensors, which alone provide insufficient information for 3D detection, to de-bias our depth estimation. We propose a depth-propagation algorithm, guided by the initial depth estimates, to diffuse these few exact measurements across the entire depth map. We show on the KITTI object detection benchmark that our combined approach yields substantial improvements in depth estimation and stereo-based 3D object detection --- outperforming the previous state-of-the-art detection accuracy for faraway objects by 40%. Our code is available at https://github.com/mileyan/Pseudo_Lidar_V2.

Motivation & Objective

• Address the depth estimation bias in stereo-based 3D detection, especially for faraway objects.
• Develop a depth-directed stereo network that learns direct depth instead of disparity.
• Combine dense stereo depth with sparse LiDAR through a graph-based diffusion to correct depths.

Proposed method

• Replace disparity-based learning with direct depth loss in a depth cost volume, enabling 3D convolutions on depth grid (SDN).
• Interpolate and transform standard disparity cost volume to a depth cost volume to reduce depth errors at long range.
• Propose a depth propagation (GDC) algorithm that uses few exact LiDAR measurements to guide diffusion of depth across a KNN graph, yielding dense accurate depths.
• Use a weighted KNN graph to propagate LiDAR depths by solving a quadratic optimization with constraints ensuring landmark depths remain fixed.
• Evaluate with stereo depth network (SDN) trained on synthetic data and KITTI; apply graph-based correction on SDN outputs.

Experimental results

Research questions

• RQ1 Can direct depth optimization in stereo networks reduce depth bias for distant objects compared to disparity-based learning?
• RQ2 Does incorporating sparse LiDAR measurements via graph-based depth correction meaningfully improve dense depth maps and 3D object detection?
• RQ3 What is the relative contribution of depth loss versus depth cost volume and the GDC step to overall detection accuracy?
• RQ4 How close can stereo-based pseudo-LiDAR come to LiDAR-based detection when augmented with sparse LiDAR?
• RQ5 Are gains consistent across car, pedestrian, and cyclist detections on KITTI?

Key findings

• SDN significantly reduces depth estimation errors at far ranges compared to disparity-based methods.
• Depth loss and depth cost volume provide measurable gains (e.g., 6%/2% and 2–3% improvements depending on the detector).
• GDC with 4-beam LiDAR substantially boosts detection, achieving near LiDAR-level performance on some metrics.
• PL++ with SDN and GDC outperforms previous stereo-based approaches by up to 40% for faraway objects on KITTI.
• With L# + S input, PL++ approaches the performance of 64-beam LiDAR on certain metrics, reducing the cost gap.
• Qualitatively, SDN+GDC yields depth maps and object localizations that align better with ground truth, especially for distant objects.