[Paper Review] Common Corruption Robustness of Point Cloud Detectors: Benchmark and Enhancement

Shuangzhi Li, Zhijie Wang+4 more|arXiv (Cornell University)|Oct 12, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper introduces the first physical-aware benchmark for evaluating point cloud object detectors under 25 common corruptions across 6 severity levels, simulating real-world degradation like rain, snow, fog, and sensor noise. Using this benchmark, the authors evaluate 8 state-of-the-art detectors and find that existing data augmentation and denoising methods offer limited robustness gains, highlighting the need for new robustness-enhancing techniques in 3D detection.

ABSTRACT

Object detection through LiDAR-based point cloud has recently been important in autonomous driving. Although achieving high accuracy on public benchmarks, the state-of-the-art detectors may still go wrong and cause a heavy loss due to the widespread corruptions in the real world like rain, snow, sensor noise, etc. Nevertheless, there is a lack of a large-scale dataset covering diverse scenes and realistic corruption types with different severities to develop practical and robust point cloud detectors, which is challenging due to the heavy collection costs. To alleviate the challenge and start the first step for robust point cloud detection, we propose the physical-aware simulation methods to generate degraded point clouds under different real-world common corruptions. Then, for the first attempt, we construct a benchmark based on the physical-aware common corruptions for point cloud detectors, which contains a total of 1,122,150 examples covering 7,481 scenes, 25 common corruption types, and 6 severities. With such a novel benchmark, we conduct extensive empirical studies on 8 state-of-the-art detectors that contain 6 different detection frameworks. Thus we get several insight observations revealing the vulnerabilities of the detectors and indicating the enhancement directions. Moreover, we further study the effectiveness of existing robustness enhancement methods based on data augmentation and data denoising. The benchmark can potentially be a new platform for evaluating point cloud detectors, opening a door for developing novel robustness enhancement methods.

Motivation & Objective

• To address the lack of a comprehensive, large-scale benchmark for evaluating point cloud detector robustness under realistic corruptions.
• To simulate physically plausible degradation in LiDAR point clouds due to weather, sensor noise, and environmental factors.
• To empirically analyze the vulnerabilities of 8 state-of-the-art 3D object detectors under diverse corruptions.
• To evaluate the effectiveness of existing robustness enhancement techniques such as data augmentation and denoising.
• To provide a new benchmark platform for future research in robust 3D object detection.

Proposed method

• Propose physical-aware simulation methods to generate realistic point cloud corruptions based on physical imaging principles of LiDAR systems.
• Construct a large-scale benchmark with 1,122,150 examples across 7,481 scenes, 25 corruption types, and 6 severity levels.
• Apply the benchmark to evaluate 8 state-of-the-art 3D detectors across 6 different detection frameworks.
• Implement two robustness enhancement strategies: physical-aware data augmentation (PA-DA) and KNN-based outlier removal (KNN-RO).
• Use mean average precision (mAP) and corruption evaluation AP (CE_AP) as metrics to quantify detector performance under corruptions.

Figure 1 : Feature visualization of the snow classification by T-SNE

Experimental results

Research questions

• RQ1 How do state-of-the-art point cloud detectors perform under a wide range of realistic, physically plausible corruptions?
• RQ2 What are the key vulnerabilities of current 3D detectors when exposed to common corruptions like rain, snow, fog, and sensor noise?
• RQ3 How effective are existing data augmentation and denoising techniques in improving detector robustness against these corruptions?
• RQ4 Are the robustness improvements consistent across different object categories (e.g., cars vs. pedestrians)?
• RQ5 Can the proposed benchmark serve as a reliable platform for evaluating and guiding future robustness enhancement research?

Key findings

• BtcDet and SE-SSD show a significant drop in detection accuracy under corruptions, with a FD increase of over 4%, indicating high vulnerability.
• KNN-based outlier removal (KNN-RO) fails to improve robustness for car detection, decreasing average CE_AP by 0.17% on average, and only slightly improving pedestrian detection.
• Physical-aware data augmentation (PA-DA) leads to a modest average CE_AP improvement of 0.16% on car detection, indicating limited effectiveness.
• The combination of PA-DA and KNN-RO increases average CE_AP to 13.38% on car detection, but this is still low, suggesting insufficient robustness gains.
• Detectors like PointRCNN and VoTr-SSD show up to 2.46% AP improvement with PA-DA, indicating some benefit for specific architectures.
• The benchmark reveals that current robustness enhancement methods are largely ineffective, calling for new, more targeted approaches to improve detector robustness.

Figure 2 : Feature visualization of the fog classification by T-SNE