[Paper Review] STD: Sparse-to-Dense 3D Object Detector for Point Cloud

Zetong Yang, Yanan Sun+3 more|arXiv (Cornell University)|Jul 22, 2019

Advanced Neural Network Applications34 references56 citations

TL;DR

STD introduces a two-stage 3D object detector that seeds point-based spherical anchors, uses a PointsPool layer to form compact proposal features, and adds a 3D IoU branch to improve localization, achieving state-of-the-art KITTI results with around 10 FPS.

ABSTRACT

We present a new two-stage 3D object detection framework, named sparse-to-dense 3D Object Detector (STD). The first stage is a bottom-up proposal generation network that uses raw point cloud as input to generate accurate proposals by seeding each point with a new spherical anchor. It achieves a high recall with less computation compared with prior works. Then, PointsPool is applied for generating proposal features by transforming their interior point features from sparse expression to compact representation, which saves even more computation time. In box prediction, which is the second stage, we implement a parallel intersection-over-union (IoU) branch to increase awareness of localization accuracy, resulting in further improved performance. We conduct experiments on KITTI dataset, and evaluate our method in terms of 3D object and Bird's Eye View (BEV) detection. Our method outperforms other state-of-the-arts by a large margin, especially on the hard set, with inference speed more than 10 FPS.

Motivation & Objective

• Motivate accurate 3D object detection directly on raw point clouds without heavy voxelization.
• Propose a point-based proposal generation with spherical anchors to preserve localization information.
• Introduce PointsPool to convert sparse proposal features into compact representations for CNN-based heads.
• Enhance box prediction with a parallel 3D IoU branch to align localization with classification.
• Demonstrate state-of-the-art performance on KITTI BEV and 3D detection, with real-time inference speeds.

Proposed method

• Seed anchors for each point using a spherical receptive field to generate high-recall proposals.
• Use a 3D semantic segmentation backbone to provide per-point features for proposal scoring.
• Apply a PointsPool layer to extract compact, differentiable proposal features from interior points.
• Employ a two-branch box predictor to estimate box parameters and predict 3D IoU for improved NMS sorting.
• Integrate a hybrid angle prediction (classification+regression) for orientation.
• Train with a multi-task loss combining segmentation, proposal classification/regression, and IoU/box losses.

Experimental results

Research questions

• RQ1 Can a point-based, spherical-anchor strategy achieve high recall while reducing anchors and computation?
• RQ2 Does transforming sparse proposal features into dense, CNN-friendly representations via PointsPool improve speed and accuracy?
• RQ3 Can a 3D IoU prediction branch improve post-processing and localization accuracy beyond traditional NMS?
• RQ4 What KITTI BEV and 3D detection gains are achievable with this sparse-to-dense two-stage approach?
• RQ5 Is the method efficient enough for real-time inference on standard GPUs?

Key findings

• Achieves state-of-the-art performance on KITTI for Car, Pedestrian, and Cyclist across BEV and 3D metrics, with notable gains on hard sets.
• Outperforms voxel- and other point-based detectors while delivering around 10 FPS inference on a TitanV GPU.
• Spherical anchors with PointsIoU labeling yield high recall with substantially fewer anchors (~16K to ~500 after pruning).
• PointsPool provides a differentiable, gradient-friendly way to convert sparse interior point features into dense proposal representations, enabling efficient FC-based heads.
• IoU estimation branch improves NMS effectiveness, yielding up to ~1.1% gains on moderate KITTI val and better alignment between classification confidence and localization quality.
• Using a combination of classification score and predicted IoU for NMS sorting (cls-score × 3D-IoU) yields better final AP than using either alone.