[Paper Review] HVPR: Hybrid Voxel-Point Representation for Single-stage 3D Object Detection

Jongyoun Noh, Sanghoon Lee+1 more|arXiv (Cornell University)|Apr 2, 2021

Advanced Neural Network Applications4 references4 citations

TL;DR

HVPR proposes a single-stage 3D object detection framework that fuses voxel-based and point-based features via a memory-augmented CNN to achieve high accuracy and efficiency. By using a memory module to store and retrieve point-based features, it enables fast inference while preserving fine-grained 3D structures, achieving 81.32% mAP on KITTI's moderate car detection with 36.1 fps.

ABSTRACT

We address the problem of 3D object detection, that is, estimating 3D object bounding boxes from point clouds. 3D object detection methods exploit either voxel-based or point-based features to represent 3D objects in a scene. Voxel-based features are efficient to extract, while they fail to preserve fine-grained 3D structures of objects. Point-based features, on the other hand, represent the 3D structures more accurately, but extracting these features is computationally expensive. We introduce in this paper a novel single-stage 3D detection method having the merit of both voxel-based and point-based features. To this end, we propose a new convolutional neural network (CNN) architecture, dubbed HVPR, that integrates both features into a single 3D representation effectively and efficiently. Specifically, we augment the point-based features with a memory module to reduce the computational cost. We then aggregate the features in the memory, semantically similar to each voxel-based one, to obtain a hybrid 3D representation in a form of a pseudo image, allowing to localize 3D objects in a single stage efficiently. We also propose an Attentive Multi-scale Feature Module (AMFM) that extracts scale-aware features considering the sparse and irregular patterns of point clouds. Experimental results on the KITTI dataset demonstrate the effectiveness and efficiency of our approach, achieving a better compromise in terms of speed and accuracy.

Motivation & Objective

• Address the trade-off between accuracy and efficiency in 3D object detection using point clouds.
• Overcome the limitations of voxel-based methods (loss of fine-grained structure) and point-based methods (high computational cost).
• Integrate the strengths of both voxel-based and point-based features into a unified, efficient representation for single-stage detection.
• Develop a memory module to reduce inference cost of point-based features without sacrificing performance.
• Enhance multi-scale feature learning for sparse and irregular point clouds using 3D scale-aware representations.

Proposed method

• Propose a two-stream encoder to extract voxel-based and point-based features separately.
• Introduce a memory module that stores and retrieves point-based features during inference, eliminating the need for real-time encoder computation.
• Aggregate memory-stored point features based on semantic similarity to voxel locations, forming a hybrid pseudo-image representation.
• Design an Attentive Multi-scale Feature Module (AMFM) that uses 3D scale features to refine multi-scale features via spatial attention.
• Use the hybrid pseudo-image as input to a single-stage detector for end-to-end 3D bounding box prediction.
• Train the model end-to-end with a standard detection head, optimizing for mAP and inference speed.

Experimental results

Research questions

• RQ1 Can a hybrid 3D representation combining voxel and point features improve detection accuracy while maintaining inference efficiency?
• RQ2 How effective is a memory module in reducing the computational cost of point-based feature extraction without degrading performance?
• RQ3 To what extent do scale-aware features enhance detection performance on sparse and irregular point clouds?
• RQ4 Does the fusion of voxel and point features lead to better localization of small or occluded objects?
• RQ5 Can the proposed method achieve a better speed-accuracy trade-off than existing single-stage 3D detectors?

Key findings

• The proposed HVPR achieves 81.32% mAP on the KITTI moderate car detection split, outperforming baseline PointPillars by 1.94% mAP gain on hard samples.
• The memory module reduces inference time from 6.8Hz (full point encoder) to 37.9Hz, nearly matching the baseline PointPillars' 39.5Hz while retaining high accuracy.
• Using 3D scale features in AMFM leads to a 1.63% mAP improvement on the hard split, demonstrating its effectiveness for complex scale variations.
• The model achieves 36.1 fps inference speed, offering a superior speed-accuracy trade-off compared to state-of-the-art methods.
• Ablation studies confirm that both the memory module and AMFM with 3D scale features are essential, with the latter contributing 1.27% mAP gain on the hard split.
• mAP performance is robust to the number of memory items (T), with optimal performance at K=20 nearest items, indicating effective feature aggregation.