[Paper Review] Back to the Feature: Classical 3D Features are (Almost) All You Need for 3D Anomaly Detection

Eliahu Horwitz, Yedid Hoshen|arXiv (Cornell University)|Mar 10, 2022

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper proposes BTF (Back to the Feature), a simple yet highly effective method for 3D anomaly detection that combines classical, rotation-invariant 3D features (FPFH) with deep color-based features (PatchCore). It achieves state-of-the-art performance on MVTec 3D-AD with 99.3% pixel-wise ROCAUC and 96.4% PRO, demonstrating that handcrafted 3D features outperform learning-based counterparts and that modality fusion is key for robust anomaly detection.

ABSTRACT

Despite significant advances in image anomaly detection and segmentation, few methods use 3D information. We utilize a recently introduced 3D anomaly detection dataset to evaluate whether or not using 3D information is a lost opportunity. First, we present a surprising finding: standard color-only methods outperform all current methods that are explicitly designed to exploit 3D information. This is counter-intuitive as even a simple inspection of the dataset shows that color-only methods are insufficient for images containing geometric anomalies. This motivates the question: how can anomaly detection methods effectively use 3D information? We investigate a range of shape representations including hand-crafted and deep-learning-based; we demonstrate that rotation invariance plays the leading role in the performance. We uncover a simple 3D-only method that beats all recent approaches while not using deep learning, external pre-training datasets, or color information. As the 3D-only method cannot detect color and texture anomalies, we combine it with color-based features, significantly outperforming previous state-of-the-art. Our method, dubbed BTF (Back to the Feature) achieves pixel-wise ROCAUC: 99.3% and PRO: 96.4% on MVTec 3D-AD.

Motivation & Objective

• To investigate whether 3D information significantly improves anomaly detection performance compared to color-only methods on 3D data.
• To identify the key properties of effective 3D representations for anomaly detection.
• To evaluate the complementary benefits of combining 3D shape and color modalities in anomaly detection.
• To establish a strong, simple baseline for future 3D anomaly detection research.

Proposed method

• The method uses the FPFH (Fast Point Feature Histograms) descriptor as a handcrafted, rotation-invariant 3D feature extractor from point clouds.
• Color-based features are extracted using PatchCore, a state-of-the-art deep learning method that leverages pre-trained features and k-NN scoring for anomaly detection.
• A late-fusion strategy combines FPFH and PatchCore features by concatenating their latent representations before anomaly scoring.
• A preprocessing pipeline removes background planes using RANSAC and connected components, improving performance for depth-based methods while slightly reducing it for FPFH.
• All features are extracted at a resolution of 28×28 patches, with average pooling used to align feature maps across different resolutions.
• Evaluation is performed on the MVTec 3D-AD dataset using standard metrics: pixel-wise ROCAUC, PRO, and image-level ROCAUC.

Experimental results

Research questions

• RQ1 Do current 3D anomaly detection methods truly outperform state-of-the-art 2D methods when applied to 3D data?
• RQ2 Is 3D information necessary for detecting geometric anomalies that are invisible in color-only images?
• RQ3 What are the key properties of effective 3D representations for anomaly detection?
• RQ4 Can combining 3D shape and color modalities yield superior performance compared to using either modality alone?

Key findings

• Color-only methods like PatchCore significantly outperform existing 3D-specific anomaly detection methods on the MVTec 3D-AD dataset.
• Rotation-invariant 3D descriptors such as FPFH achieve superior performance compared to both deep learning-based 3D representations and non-invariant classical features.
• The proposed BTF method, combining FPFH and PatchCore, achieves a new state-of-the-art with 99.3% pixel-wise ROCAUC and 96.4% PRO on MVTec 3D-AD.
• Preprocessing that removes background planes improves performance for depth-based methods but slightly degrades results for FPFH, suggesting differing sensitivity to missing data.
• Despite high pixel-level performance, image-level detection accuracy (I-ROC) remains suboptimal at 86.5%, indicating a need for better image-level scoring metrics for 3D data.
• Certain classes like cable gland and foam remain challenging for 3D-based methods, and fusion with color features does not fully resolve the performance gap, indicating a limitation in current fusion strategies.