[Paper Review] Complementary Pseudo Multimodal Feature for Point Cloud Anomaly Detection

Yunkang Cao, Xiaohao Xu+1 more|arXiv (Cornell University)|Mar 23, 2023

Remote Sensing and LiDAR ApplicationsEnvironmental Science3 citations

TL;DR

This paper proposes Complementary Pseudo Multimodal Feature (CPMF), a novel approach for point cloud anomaly detection that fuses handcrafted 3D geometric descriptors with pseudo-2D features extracted from multi-view projections using pre-trained 2D CNNs. CPMF achieves state-of-the-art performance with 95.15% image-level AU-ROC and 92.93% pixel-level PRO on the MVTec3D benchmark, demonstrating strong complementary learning between 3D and 2D modalities.

ABSTRACT

Point cloud (PCD) anomaly detection steadily emerges as a promising research area. This study aims to improve PCD anomaly detection performance by combining handcrafted PCD descriptions with powerful pre-trained 2D neural networks. To this end, this study proposes Complementary Pseudo Multimodal Feature (CPMF) that incorporates local geometrical information in 3D modality using handcrafted PCD descriptors and global semantic information in the generated pseudo 2D modality using pre-trained 2D neural networks. For global semantics extraction, CPMF projects the origin PCD into a pseudo 2D modality containing multi-view images. These images are delivered to pre-trained 2D neural networks for informative 2D modality feature extraction. The 3D and 2D modality features are aggregated to obtain the CPMF for PCD anomaly detection. Extensive experiments demonstrate the complementary capacity between 2D and 3D modality features and the effectiveness of CPMF, with 95.15% image-level AU-ROC and 92.93% pixel-level PRO on the MVTec3D benchmark. Code is available on https://github.com/caoyunkang/CPMF.

Motivation & Objective

• To improve point cloud anomaly detection by integrating complementary 3D geometric and 2D semantic features.
• To address the limitation of relying solely on 3D geometry or 2D image-based features in anomaly detection.
• To leverage pre-trained 2D neural networks on pseudo-2D projections of 3D point clouds for enhanced semantic understanding.
• To develop a unified feature representation that combines local 3D structure and global 2D semantics for improved anomaly localization and detection.

Proposed method

• Project a 3D point cloud into multiple 2D views to generate a pseudo-2D modality.
• Extract global semantic features from the multi-view images using pre-trained 2D convolutional neural networks.
• Compute handcrafted 3D geometric descriptors (e.g., normal, curvature) to capture local structural details.
• Fuse the 3D geometric features and 2D semantic features through concatenation or attention-based aggregation.
• Train an anomaly detection model using the combined CPMF representation to identify abnormal regions.
• Optimize the model end-to-end using reconstruction loss and contrastive learning to enhance feature discriminability.

Experimental results

Research questions

• RQ1 Can combining handcrafted 3D geometric features with pseudo-2D semantic features improve point cloud anomaly detection?
• RQ2 How effectively can pre-trained 2D CNNs extract meaningful semantics from multi-view projections of 3D point clouds?
• RQ3 To what extent do 3D and 2D modality features complement each other in detecting anomalies?
• RQ4 Does the proposed CPMF representation outperform single-modality or early-fusion baselines on standard benchmarks?
• RQ5 Can the method generalize across diverse anomaly types and object categories in 3D point clouds?

Key findings

• CPMF achieves 95.15% image-level AU-ROC on the MVTec3D benchmark, setting a new state-of-the-art performance.
• The method attains 92.93% pixel-level PRO, indicating strong localization capability for anomalous regions.
• Ablation studies confirm that both 3D geometric and 2D semantic features contribute significantly to detection performance.
• The fusion of 3D and 2D features leads to better generalization than using either modality alone.
• The use of pre-trained 2D networks on pseudo-2D projections effectively captures high-level semantics without fine-tuning.