[Paper Review] Benchmarking Robustness of 3D Point Cloud Recognition Against Common Corruptions

J. F. Sun, Qingzhao Zhang+4 more|arXiv (Cornell University)|Jan 28, 2022

Industrial Vision Systems and Defect DetectionEngineering49 citations

TL;DR

Introduces ModelNet40-C, the first comprehensive corruption-robustness benchmark for 3D point cloud recognition, evaluates six architectures under 15 corruptions, and shows simple augmentation (PointCutMix-R) plus test-time adaptation (TENT) substantially improve robustness.

ABSTRACT

Deep neural networks on 3D point cloud data have been widely used in the real world, especially in safety-critical applications. However, their robustness against corruptions is less studied. In this paper, we present ModelNet40-C, the first comprehensive benchmark on 3D point cloud corruption robustness, consisting of 15 common and realistic corruptions. Our evaluation shows a significant gap between the performances on ModelNet40 and ModelNet40-C for state-of-the-art (SOTA) models. To reduce the gap, we propose a simple but effective method by combining PointCutMix-R and TENT after evaluating a wide range of augmentation and test-time adaptation strategies. We identify a number of critical insights for future studies on corruption robustness in point cloud recognition. For instance, we unveil that Transformer-based architectures with proper training recipes achieve the strongest robustness. We hope our in-depth analysis will motivate the development of robust training strategies or architecture designs in the 3D point cloud domain. Our codebase and dataset are included in https://github.com/jiachens/ModelNet40-C

Motivation & Objective

• Motivate the study of corruption robustness for 3D point cloud recognition in safety-critical applications.
• Create a systematic corruption benchmark (ModelNet40-C) with realistic, diverse corruptions.
• Evaluate representative architectures on ModelNet40-C to identify robustness gaps.
• Identify effective strategies (augmentation and test-time adaptation) to reduce performance gaps under corruption.
• Provide actionable insights to guide robust architecture and training design for point clouds.

Proposed method

• Design 15 corruption types grouped into density, noise, and transformation categories, each with five severity levels.
• Construct ModelNet40-C by applying corruptions to the ModelNet40 validation set and ensuring semantic preservation.
• Evaluate six representative architectures (PointNet, PointNet++, DGCNN, RSCNN, PCT, SimpleView) under standard training.
• Survey a range of data augmentation and test-time adaptation methods (PointCutMix-R, PointCutMix-K, PointMixup, RSMix, PGD adversarial training; BN, TENT) across 3,180 configurations.
• Analyze results to derive architectural and training-time insights for corruption robustness.

Experimental results

Research questions

• RQ1 How robust are common 3D point cloud architectures to realistic corruptions beyond clean ModelNet40 scenarios?
• RQ2 Which corruption types most challenge current architectures, and how does robustness vary by architecture?
• RQ3 Can data augmentation and test-time adaptation substantially reduce corruption-induced performance gaps?
• RQ4 Which architectural designs (e.g., Transformer-based) inherently confer robustness to specific corruption families?
• RQ5 What practical strategies best improve corruption robustness for 3D point cloud recognition?

Key findings

• ModelNet40-C reveals about a 3x error-rate gap between ModelNet40 and ModelNet40-C for SOTA models.
• Occlusion, LiDAR, rotation, and background noise are particularly challenging across architectures.
• Transformer-based architectures (PCT) show strongest robustness to transformation-type corruptions.
• Different architectures vary in vulnerability; PointNet is strong on density but weak overall, while Ball-query-based methods offer robustness to Background noise.
• Data augmentation generally improves corruption robustness, with PointCutMix-R achieving the best overall robustness (ER_corrup = 18.7%), though no single augmentation dominates all corruptions.
• Test-time adaptations (BN, TENT) help but are generally outperformed by data augmentation strategies.