[Paper Review] StarNet: Targeted Computation for Object Detection in Point Clouds

Jiquan Ngiam, Benjamin Caine+11 more|arXiv (Cornell University)|Aug 29, 2019

Advanced Neural Network Applications51 references100 citations

TL;DR

StarNet is a sparse, point-based LiDAR detector that uses data-dependent proposals and local featurization to achieve competitive 3D object detection while enabling dynamic control of inference cost and temporal contextualization.

ABSTRACT

Detecting objects from LiDAR point clouds is an important component of self-driving car technology as LiDAR provides high resolution spatial information. Previous work on point-cloud 3D object detection has re-purposed convolutional approaches from traditional camera imagery. In this work, we present an object detection system called StarNet designed specifically to take advantage of the sparse and 3D nature of point cloud data. StarNet is entirely point-based, uses no global information, has data dependent anchors, and uses sampling instead of learned region proposals. We demonstrate how this design leads to competitive or superior performance on the large Waymo Open Dataset and the KITTI detection dataset, as compared to convolutional baselines. In particular, we show how our detector can outperform a competitive baseline on Pedestrian detection on the Waymo Open Dataset by more than 7 absolute mAP while being more computationally efficient. We show how our redesign---namely using only local information and using sampling instead of learned proposals---leads to a significantly more flexible and adaptable system: we demonstrate how we can vary the computational cost of a single trained StarNet without retraining, and how we can target proposals towards areas of interest with priors and heuristics. Finally, we show how our design allows for incorporating temporal context by using detections from previous frames to target computation of the detector, which leads to further improvements in performance without additional computational cost.

Motivation & Objective

• Motivate detectors that exploit the sparsity and 3D nature of LiDAR data for real-time autonomous driving.
• Develop a fully point-based detector that processes proposals independently without global context.
• Introduce cheap, data-driven center sampling and local point cloud featurization to replace learned region proposals.
• Show that computation can be dynamically allocated at inference time by varying proposals and points per proposal.
• Demonstrate that temporal context from previous frames can target computation to improve detection without extra cost.

Proposed method

• Propose StarNet, a local, non-global detector that samples centers from the LiDAR point cloud and processes each center independently.
• Use random or farthest-point sampling (FPS) to generate proposal centers within z-bounds, optionally seeded by previous frame detections.
• Featurize a local point cloud around each proposal with a stack of StarNet blocks that aggregate local/global statistics and produce a 384-d feature per proposal.
• Construct final predictions by applying a data-dependent grid of anchors around each center, projecting from the 384-d feature to classification and regression logits for bounding box parameters.
• Train with focal loss for classification and SmoothL1 losses for bounding box regression; use IoU-based assignment with a positive/negative threshold scheme.
• Leverage temporal context by seeding proposal centers with high-confidence detections from previous frames to improve current-frame detection without retraining.

Experimental results

Research questions

• RQ1 Can a fully point-based detector that operates on local, non-aggregated proposals compete with convolution-based methods for 3D object detection?
• RQ2 Does exploiting LiDAR sparsity and data-dependent sampling enable flexible control over inference cost without retraining?
• RQ3 How does incorporating temporal context from prior frames affect detection performance and computation?
• RQ4 What is the impact of different center-sampling strategies (random vs FPS vs temporal seeding) on detection coverage and accuracy?

Key findings

• StarNet achieves competitive 3D detection performance on KITTI and Waymo Open Dataset compared to convolutional baselines.
• On Waymo Pedestrian detection, StarNet outperforms a competitive baseline by more than 7 absolute mAP at similar or lower compute.
• Temporal context using previous frame detections can substantially boost mAP, e.g., up to ~40% relative improvement in Pedestrian mAP when seeding centers with prior detections.
• Using FPS sampling yields more uniform spatial coverage and better center proposals than random sampling, especially at fixed proposal budgets.
• A single trained StarNet model can adapt its computational cost by varying the number of proposals or the number of points per proposal without retraining, achieving substantial efficiency gains.
• With temporal seeding, using 32 prior detections in 384 centers raises Pedestrian mAP from 41.8 to 53.2 on validation, and 192 prior detections raise it to 58.0 (vs 384 centers); using 512 prior detections with 1024 centers yields ~69.7 mAP.