[Paper Review] SIENet: Spatial Information Enhancement Network for 3D Object Detection from Point Cloud

Ziyu Li, Yuncong Yao+3 more|arXiv (Cornell University)|Mar 29, 2021

Advanced Neural Network Applications45 references17 citations

TL;DR

SIENet proposes a two-stage 3D object detection framework that addresses point cloud density imbalance in LiDAR data by introducing a Spatial Information Enhancement (SIE) module to predict dense, complete shapes within candidate proposals, and a Hybrid-Paradigm Region Proposal Network (HP-RPN) for robust feature extraction. The method achieves state-of-the-art performance on KITTI, with a 4.39 AP gain at far distances (beyond 20m), significantly improving detection robustness in sparse, long-range scenarios.

ABSTRACT

LiDAR-based 3D object detection pushes forward an immense influence on autonomous vehicles. Due to the limitation of the intrinsic properties of LiDAR, fewer points are collected at the objects farther away from the sensor. This imbalanced density of point clouds degrades the detection accuracy but is generally neglected by previous works. To address the challenge, we propose a novel two-stage 3D object detection framework, named SIENet. Specifically, we design the Spatial Information Enhancement (SIE) module to predict the spatial shapes of the foreground points within proposals, and extract the structure information to learn the representative features for further box refinement. The predicted spatial shapes are complete and dense point sets, thus the extracted structure information contains more semantic representation. Besides, we design the Hybrid-Paradigm Region Proposal Network (HP-RPN) which includes multiple branches to learn discriminate features and generate accurate proposals for the SIE module. Extensive experiments on the KITTI 3D object detection benchmark show that our elaborately designed SIENet outperforms the state-of-the-art methods by a large margin.

Motivation & Objective

• Address the challenge of imbalanced point cloud density in LiDAR-based 3D object detection, particularly the degradation in performance at long distances due to sparse point sampling.
• Improve feature representation for 3D object detection by enhancing spatial structure information from incomplete, sparse point sets within candidate proposals.
• Develop a robust region proposal network that effectively extracts multi-scale features and generates high-recall proposals even under density imbalance.
• Design a spatial information enhancement mechanism that predicts dense, complete shapes from sparse foreground points to enrich semantic and structural features for box refinement.

Proposed method

• Proposes the Spatial Information Enhancement (SIE) module to predict complete, dense spatial shapes from sparse point sets within 3D object proposals, using a learnable shape generation head.
• Introduces a Hybrid-Paradigm Region Proposal Network (HP-RPN) with three branches: a spconv-based branch for multi-scale feature extraction, an auxiliary branch to guide spatial learning, and a keypoint branch for dynamic feature reweighting.
• Employs attention-based feature fusion in the SIE module to better aggregate enhanced spatial features with original proposal features for improved representation.
• Utilizes a two-stage detection pipeline: HP-RPN generates proposals in stage one, followed by SIE and box refinement in stage two.
• Leverages sparse 3D convolution and point-based feature learning to maintain efficiency and accuracy in processing large-scale, unordered point clouds.
• Applies context fusion between the auxiliary and spconv branches in HP-RPN to enhance feature discrimination and proposal quality.

Experimental results

Research questions

• RQ1 Can predicting complete, dense spatial shapes from sparse point sets improve 3D object detection performance in long-range, low-density scenarios?
• RQ2 How effective is a hybrid-paradigm RPN design with multiple specialized branches in handling imbalanced point cloud density and improving proposal quality?
• RQ3 To what extent does the integration of structure-aware spatial information enhance feature representation and box refinement in 3D detectors?
• RQ4 Does the attention-based fusion mechanism in the SIE module lead to better feature aggregation than simple concatenation?

Key findings

• SIENet achieves a 3.23 average AP gain across all distance ranges on the KITTI 3D detection benchmark compared to prior state-of-the-art methods.
• The method achieves a 5.13 AP gain at medium distances and a 4.39 AP gain at far distances (beyond 20m), demonstrating superior robustness to sparse point clouds.
• The ablation study confirms that the SIE module with attention-based fusion yields the highest performance gain, improving moderate and hard difficulty AP by 0.84% and 0.42% respectively.
• The HP-RPN with both auxiliary and context fusion components achieves the best performance, showing a 0.54% AP gain on easy, 0.95% on moderate, and 0.38% on hard examples.
• Removing the SIE module leads to a significant drop in performance, confirming its critical role in enhancing feature representation for box refinement.
• The model maintains competitive performance at near distances (AP ~92) while drastically outperforming others at far distances, validating its focus on long-range detection.