[Paper Review] LiDAR Cluster First and Camera Inference Later: A New Perspective Towards Autonomous Driving

Jiyang Chen, Simon C.H. Yu+5 more|arXiv (Cornell University)|Nov 18, 2021

Advanced Neural Network Applications15 references4 citations

TL;DR

This paper proposes a novel end-to-end autonomous driving perception pipeline that prioritizes high-risk objects by first clustering LiDAR point clouds into 3D clusters, projecting them to 2D camera regions (2D clusters), and then applying camera-based inference only on these prioritized regions. The method reduces priority inversion, improves inference speed by 25% compared to full-frame processing, and maintains comparable detection accuracy using the Waymo Open Dataset.

ABSTRACT

Object detection in state-of-the-art Autonomous Vehicles (AV) framework relies heavily on deep neural networks. Typically, these networks perform object detection uniformly on the entire camera LiDAR frames. However, this uniformity jeopardizes the safety of the AV by giving the same priority to all objects in the scenes regardless of their risk of collision to the AV. In this paper, we present a new end-to-end pipeline for AV that introduces the concept of LiDAR cluster first and camera inference later to detect and classify objects. The benefits of our proposed framework are twofold. First, our pipeline prioritizes detecting objects that pose a higher risk of collision to the AV, giving more time for the AV to react to unsafe conditions. Second, it also provides, on average, faster inference speeds compared to popular deep neural network pipelines. We design our framework using the real-world datasets, the Waymo Open Dataset, solving challenges arising from the limitations of LiDAR sensors and object detection algorithms. We show that our novel object detection pipeline prioritizes the detection of higher risk objects while simultaneously achieving comparable accuracy and a 25% higher average speed compared to camera inference only.

Motivation & Objective

• To address priority inversion in autonomous vehicle perception systems, where all objects are processed uniformly despite varying collision risk.
• To reduce computational waste by focusing object detection on only the most critical regions of the camera frame.
• To design a real-world compatible pipeline that avoids reliance on idealized bounding boxes or perfect LiDAR annotations.
• To achieve faster inference speeds without sacrificing detection accuracy in real-time autonomous driving applications.
• To enable efficient GPU utilization through depth-aware composite image grouping and optimization.

Proposed method

• Using the Depth Clustering algorithm on raw LiDAR point clouds to segment 3D objects based on depth discontinuities.
• Projecting 3D clusters onto 2D camera images to generate 2D clusters with associated depth values.
• Applying a depth-aware merging algorithm to combine overlapping or scattered 2D clusters into Collision Avoidance Zones (CAZones).
• Designing a depth-aware First-Fit Decreasing Height (FFDH) algorithm to group CAZones into composite images for efficient GPU batching.
• Optimizing the pipeline with GPU profiling and depth-based prioritization to bound priority inversion and reduce end-to-end latency.
• Evaluating the system on the Waymo Open Dataset using YOLOv3 for inference on composite CAZones.

Experimental results

Research questions

• RQ1 Can a LiDAR-first, camera-inference-later pipeline reduce priority inversion in autonomous vehicle perception by focusing on high-risk regions?
• RQ2 How does the proposed method compare to full-frame camera inference in terms of inference speed and detection accuracy?
• RQ3 Can real-world LiDAR data be effectively used to generate reliable, depth-aware regions of interest without relying on perfect bounding box assumptions?
• RQ4 To what extent can depth-aware merging and composite image grouping improve GPU utilization and reduce end-to-end latency?
• RQ5 Does the pipeline maintain or improve detection accuracy despite processing only a subset of the camera frame?

Key findings

• The proposed pipeline achieved an average inference time of 129 ms, which is 25% faster than the full-frame YOLOv3-608 baseline (197 ms).
• The LiDAR module processed point clouds in 5.7 ms on average, and the Camera module's non-inference components added only 9 ms overhead.
• The method maintained comparable detection accuracy, with an AP_{M} of 92.73%—higher than the full_608 baseline (90.75%)—and AP_{50} of 96.25%.
• The depth-aware merging algorithm successfully reduced false positives and over-segmented clusters, forming coherent CAZones for effective inference.
• The composite image grouping strategy enabled efficient GPU batching, contributing to the overall speedup without requiring DNN architecture changes.
• The system demonstrated robustness in real-world conditions using the Waymo Open Dataset, with no reliance on idealized or synthetic LiDAR annotations.