[Paper Review] Pillar R-CNN for Point Cloud 3D Object Detection

Guangsheng Shi, Ruifeng Li+1 more|arXiv (Cornell University)|Feb 26, 2023

Advanced Neural Network Applications4 citations

TL;DR

Pillar R-CNN proposes a simple yet effective two-stage 3D object detection framework that leverages pillar-based bird's-eye-view (BEV) feature representation instead of point-voxel feature interactions. By integrating FPN into the RPN for multi-scale proposal generation and using dense, lateral connections for feature fusion, it achieves state-of-the-art performance on the Waymo Open Dataset with minimal computational overhead, demonstrating that pillar-based BEV features alone can preserve sufficient 3D structure for high-accuracy detection.

ABSTRACT

The performance of point cloud 3D object detection hinges on effectively representing raw points, grid-based voxels or pillars. Recent two-stage 3D detectors typically take the point-voxel-based R-CNN paradigm, i.e., the first stage resorts to the 3D voxel-based backbone for 3D proposal generation on bird-eye-view (BEV) representation and the second stage refines them via the intermediate point representation. Their primary mechanisms involve the utilization of intermediary keypoints to restore the substantial 3D structure context from the converted BEV representation. The skilled point-voxel feature interaction, however, makes the entire detection pipeline more complex and compute-intensive. In this paper, we take a different viewpoint -- the pillar-based BEV representation owns sufficient capacity to preserve the 3D structure. In light of the latest advances in BEV-based perception, we devise a conceptually simple yet effective two-stage 3D detection architecture, named Pillar R-CNN. On top of densified BEV feature maps, Pillar R-CNN can easily introduce the feature pyramid architecture to generate 3D proposals at various scales and take the simple 2D R-CNN style detect head for box refinement. Our Pillar R-CNN performs favorably against state-of-the-art 3D detectors on the large-scale Waymo Open Dataset but at a small extra cost. It should be highlighted that further exploration into BEV perception for applications involving autonomous driving is now possible thanks to the effective and elegant Pillar R-CNN architecture.

Motivation & Objective

• To simplify two-stage 3D object detection by eliminating complex point-voxel feature interactions.
• To investigate whether pillar-based BEV representation alone can preserve sufficient 3D structural context for high-accuracy detection.
• To bridge the domain gap between 2D image detection and 3D point cloud detection by adapting 2D detection advances to BEV-based 3D detection.
• To reduce computational complexity while maintaining or improving detection accuracy compared to existing point-voxel-based two-stage detectors.

Proposed method

• Uses a pillar-based feature backbone with hybrid sparse and dense convolutions to generate hierarchical BEV feature maps in a single forward pass.
• Introduces a lateral connection layer to build a pillar-based feature pyramid, densifying sparse pillar volumes for multi-scale feature fusion.
• Employs FPN-style feature aggregation in the RPN to generate class-specific 3D proposals at multiple scales, improving detection of small objects.
• Applies RoIAlign to crop dense, 2D pooling feature maps for box refinement, using a single, manageable resolution (4× downsampled).
• Uses a class-agnostic, single-scale pooling map to improve robustness to object scale variation.
• Applies IoU-aware confidence rectification to improve proposal ranking and classification score alignment.

Experimental results

Research questions

• RQ1 Can pillar-based BEV representation alone preserve sufficient 3D structural context for high-accuracy 3D object detection?
• RQ2 Does replacing point-voxel feature interaction with a simpler pillar-based feature pyramid improve detection performance and efficiency?
• RQ3 Can FPN be effectively adapted to the RPN in a 3D detection pipeline to improve small object detection?
• RQ4 Does IoU-aware confidence rectification enhance proposal quality in a BEV-based two-stage 3D detector?

Key findings

• Pillar R-CNN achieves 76.48% AP for vehicles and 68.54% for pedestrians on the Waymo Open Dataset validation set, matching or exceeding state-of-the-art two-stage detectors.
• The model achieves comparable accuracy to PV-RCNN and Voxel R-CNN while using a simpler pipeline without intermediate keypoint abstraction.
• Incorporating FPN into the RPN improves small object detection, with the best performance achieved using the 4× downsampled feature map.
• The use of additional bottom-up features in the pooling map yields negligible gains at extra cost, indicating that the 4× strided map alone is sufficient.
• IoU-aware confidence rectification significantly improves RPN performance, enabling it to match R-CNN-level accuracy in proposal quality.
• End-to-end training of the RPN and R-CNN yields better performance than separate training, which risks overfitting or suboptimal convergence.