[Paper Review] Reviewing 3D Object Detectors in the Context of High-Resolution 3+1D Radar

Patrick Palmer, Martin Krueger+3 more|arXiv (Cornell University)|Aug 10, 2023

Advanced SAR Imaging Techniques4 citations

TL;DR

This paper evaluates ten 3D object detectors originally designed for lidar point clouds on high-resolution 3+1D radar point clouds, adapting them to radar data and assessing performance across two datasets. Key findings show no single model outperforms others consistently, with PV-RCNN and Voxel R-CNN showing stronger robustness, while PointRCNN performs poorly; noise and sparsity in radar data significantly degrade detection accuracy, indicating a need for radar-specific architectural adaptations to close the performance gap with lidar.

ABSTRACT

Recent developments and the beginning market introduction of high-resolution imaging 4D (3+1D) radar sensors have initialized deep learning-based radar perception research. We investigate deep learning-based models operating on radar point clouds for 3D object detection. 3D object detection on lidar point cloud data is a mature area of 3D vision. Many different architectures have been proposed, each with strengths and weaknesses. Due to similarities between 3D lidar point clouds and 3+1D radar point clouds, those existing 3D object detectors are a natural basis to start deep learning-based 3D object detection on radar data. Thus, the first step is to analyze the detection performance of the existing models on the new data modality and evaluate them in depth. In order to apply existing 3D point cloud object detectors developed for lidar point clouds to the radar domain, they need to be adapted first. While some detectors, such as PointPillars, have already been adapted to be applicable to radar data, we have adapted others, e.g., Voxel R-CNN, SECOND, PointRCNN, and PV-RCNN. To this end, we conduct a cross-model validation (evaluating a set of models on one particular data set) as well as a cross-data set validation (evaluating all models in the model set on several data sets). The high-resolution radar data used are the View-of-Delft and Astyx data sets. Finally, we evaluate several adaptations of the models and their training procedures. We also discuss major factors influencing the detection performance on radar data and propose possible solutions indicating potential future research avenues.

Motivation & Objective

• To assess the transferability of existing lidar-based 3D object detectors to high-resolution 3+1D radar point clouds.
• To identify performance bottlenecks in radar-based 3D detection, particularly due to data sparsity and high noise levels.
• To evaluate model generalization across diverse radar datasets, including View-of-Delft and Astyx.
• To propose improvements through data accumulation techniques and radar-specific architectural extensions.
• To establish a benchmark for future research in radar-only 3D object detection for automated driving.

Proposed method

• Adapted ten state-of-the-art 3D object detectors—point-based (PointRCNN), voxel-based (Voxel R-CNN), pillar-based (PointPillars), and point-voxel hybrid (PV-RCNN)—to operate on 3+1D radar point clouds.
• Modified network backbones and feature encoders, including extending the Voxel Feature Encoder (VFE) to handle radar-specific inputs like radial velocity.
• Trained models on the View-of-Delft (VoD) dataset, a high-resolution 3+1D radar benchmark with urban traffic scenes.
• Fine-tuned the best-performing models from VoD on the smaller Astyx dataset to evaluate cross-dataset generalization.
• Applied cross-model and cross-dataset validation to compare detection performance across models and data distributions.
• Used qualitative analysis and error mode visualization to diagnose false positives and misalignment issues caused by radar noise and sparsity.

Figure 1 : Overview of radar sensor data processing pipeline. See [ 5 , 43 , 27 ] for a more detailed description. ADC: Analog-to-Digital Converter, FFT: Fast Fourier Transformation, DC: Data Conversion, Demo: Demodulation, PD: Peak Detection

Experimental results

Research questions

• RQ1 How well do existing lidar-based 3D object detectors generalize to high-resolution 3+1D radar point cloud data?
• RQ2 What are the primary performance-limiting factors in radar-based 3D detection, and how do they compare to lidar-based detection?
• RQ3 Which 3D object detector architecture performs most robustly across different radar data distributions and object classes?
• RQ4 To what extent does radar data sparsity and high noise level degrade detection accuracy compared to lidar?
• RQ5 What data processing or architectural modifications can improve radar-only 3D object detection performance?

Key findings

• PV-RCNN and Voxel R-CNN achieve the highest and most consistent 3D mAP performance across both the View-of-Delft and Astyx datasets, indicating greater robustness to radar data characteristics.
• PointRCNN performs significantly worse than other models on radar data, making it unsuitable for direct application without major architectural modifications.
• The performance gap between lidar and radar perception cannot be attributed solely to sparsity; high noise levels in radar point clouds lead to significant misalignment and false positives.
• False positives are prevalent in sparse regions where ground truth annotations contain few points, suggesting detectors learn to predict detections even in low-signal areas.
• Naive point cloud accumulation over time leads to smearing artifacts from dynamic objects, which can be mitigated by motion compensation using radial velocity estimates.
• Radar-specific architectural extensions—such as incorporating radial velocity, self-attention mechanisms, or uncertainty estimation—show strong potential to close the performance gap with lidar.

Figure 2 : General trends for the detection performances of all the investigated models for both ranges on lidar (solid lines) and radar (dashed lines) data. PP: PointPillars, SD: SECOND, Point: Point-RCNN, PV: PV-RCNN, Voxel: Voxel R-CNN, CP: CenterPoint