[Paper Review] Center-based 3D Object Detection and Tracking

Tianwei Yin, Xingyi Zhou+1 more|arXiv (Cornell University)|Jun 19, 2020

Advanced Neural Network ApplicationsComputer Science64 references67 citations

TL;DR

CenterPoint represents, detects, and tracks 3D objects as centers (points) rather than axis-aligned boxes, enabling simple, fast, and state-of-the-art 3D detection and tracking on LiDAR data. A two-stage refinement further boosts performance.

ABSTRACT

Three-dimensional objects are commonly represented as 3D boxes in a point-cloud. This representation mimics the well-studied image-based 2D bounding-box detection but comes with additional challenges. Objects in a 3D world do not follow any particular orientation, and box-based detectors have difficulties enumerating all orientations or fitting an axis-aligned bounding box to rotated objects. In this paper, we instead propose to represent, detect, and track 3D objects as points. Our framework, CenterPoint, first detects centers of objects using a keypoint detector and regresses to other attributes, including 3D size, 3D orientation, and velocity. In a second stage, it refines these estimates using additional point features on the object. In CenterPoint, 3D object tracking simplifies to greedy closest-point matching. The resulting detection and tracking algorithm is simple, efficient, and effective. CenterPoint achieved state-of-the-art performance on the nuScenes benchmark for both 3D detection and tracking, with 65.5 NDS and 63.8 AMOTA for a single model. On the Waymo Open Dataset, CenterPoint outperforms all previous single model method by a large margin and ranks first among all Lidar-only submissions. The code and pretrained models are available at https://github.com/tianweiy/CenterPoint.

Motivation & Objective

• Motivate a center-based representation to overcome orientation and box fitting challenges in 3D detection.
• Propose a two-stage CenterPoint detector that detects centers and regresses to full 3D pose, size, and velocity.
• Introduce a lightweight second-stage refinement using point-features on object faces.
• Demonstrate end-to-end 3D detection and tracking with a simple greedy closest-point association.
• Show state-of-the-art results on Waymo Open Dataset and nuScenes with efficient inference.

Proposed method

• Use a standard 3D backbone (VoxelNet or PointPillars) to produce a map-view feature map M from LiDAR data.
• Predict a class-specific center heatmap, object size, sub-voxel localization, yaw angle, and velocity at each center via dense heads.
• Train with a CenterNet-inspired keypoint loss and regression heads for all object attributes.
• Refine first-stage predictions with a second stage that samples center-face point features from M and processes them with an MLP to output a confidence score and box refinement.
• Track by greedily matching current detections to previous frames using the predicted 2D velocity to project centers and closest-point association.

Experimental results

Research questions

• RQ1 Can a center-based representation simplify 3D object detection and tracking compared to anchor-based bounding boxes in LiDAR data?
• RQ2 Does a two-stage center-point refinement improve 3D localization, orientation, and velocity estimation with minimal overhead?
• RQ3 How well does center-based detection support real-time 3D tracking without heavy motion models?
• RQ4 What is the performance impact of center-based detection on large-scale datasets like Waymo and nuScenes?
• RQ5 How does CenterPoint compare to anchor-based methods in terms of robustness to rotation and object size variations?

Key findings

• Center-based CenterPoint improves 3D detection by replacing axis-aligned boxes with center points, yielding substantial mAP/mAPH gains across backbones and datasets.
• Two-stage CenterPoint with additional center-face features provides further performance gains with modest overhead (2-stage refinement adds about 6–7 ms in some setups).
• CenterPoint achieves state-of-the-art results on Waymo (vehicle mAPH and pedestrian mAPH at Level 2) and nuScenes (NDS and mAP), outperforming prior methods.
• CenterPoint tracking via velocity-aware greedy closest-point matching significantly improves AMOTA over Kalman-filter baselines, with up to 8.8 AMOTA gains on nuScenes and substantial improvements on Waymo.
• CenterPoint runs near real-time (e.g., 11 FPS on Waymo, 16 FPS on nuScenes) and maintains strong performance across small and highly rotated objects.