[Paper Review] Segment Anything in Defect Detection

Bozhen Hu, Bin Gao+3 more|arXiv (Cornell University)|Nov 17, 2023

Industrial Vision Systems and Defect Detection4 citations

TL;DR

This paper proposes DefectSAM, a novel foundation model for segmenting defects in highly noisy infrared thermal images using the Segment Anything (SAM) framework fine-tuned on a large-scale, expert-annotated thermal defect dataset. By leveraging expert-provided prompts and domain-specific pre-training, DefectSAM achieves state-of-the-art performance, attaining 100% F-score and 78.58% IOU on flat-type and 81.98% IOU on R-type (irregular) thermal images, significantly outperforming existing methods in detecting small, weak, and irregularly shaped defects.

ABSTRACT

Defect detection plays a crucial role in infrared non-destructive testing systems, offering non-contact, safe, and efficient inspection capabilities. However, challenges such as low resolution, high noise, and uneven heating in infrared thermal images hinder comprehensive and accurate defect detection. In this study, we propose DefectSAM, a novel approach for segmenting defects on highly noisy thermal images based on the widely adopted model, Segment Anything (SAM)\cite{kirillov2023segany}. Harnessing the power of a meticulously curated dataset generated through labor-intensive lab experiments and valuable prompts from experienced experts, DefectSAM surpasses existing state-of-the-art segmentation algorithms and achieves significant improvements in defect detection rates. Notably, DefectSAM excels in detecting weaker and smaller defects on complex and irregular surfaces, reducing the occurrence of missed detections and providing more accurate defect size estimations. Experimental studies conducted on various materials have validated the effectiveness of our solutions in defect detection, which hold significant potential to expedite the evolution of defect detection tools, enabling enhanced inspection capabilities and accuracy in defect identification.

Motivation & Objective

• To address the challenge of detecting small, weak, and irregularly shaped defects in highly noisy infrared thermal images from non-destructive testing.
• To overcome limitations of existing deep learning models in thermal image segmentation, including overfitting and poor generalization on complex surfaces.
• To develop a universal, adaptable segmentation model that generalizes across diverse materials and defect types without requiring task-specific retraining.
• To improve defect size estimation accuracy and reduce missed detections in real-world industrial inspection scenarios.
• To establish a benchmark dataset and model for thermal defect detection, enabling future research and tool development.

Proposed method

• Fine-tuning the pre-trained Segment Anything Model (SAM) on a large-scale, expert-annotated dataset of thermal images collected from optical pulsed thermography (OPT) and portable OPT (POPT) systems.
• Incorporating expert-provided prompts to guide the segmentation process, enhancing detection of subtle and irregular defects.
• Utilizing a two-stage training paradigm: pre-training on natural images followed by domain-specific fine-tuning on thermal defect data.
• Applying prompt-guided inference to improve segmentation accuracy, especially for low-contrast and noisy regions.
• Employing a multi-material, multi-geometry dataset to ensure robustness across diverse industrial surfaces and defect types.
• Validating the model using standard metrics such as F-score and Intersection over Union (IOU) on both flat and irregular (R-type) defect geometries.

Experimental results

Research questions

• RQ1 Can a foundation model like SAM be effectively adapted to detect defects in highly noisy thermal images from non-destructive testing?
• RQ2 How does expert-provided prompting improve segmentation accuracy for weak and irregularly shaped defects in thermal images?
• RQ3 To what extent does DefectSAM generalize across different materials and defect geometries compared to existing SOTA models?
• RQ4 Can a single model achieve superior performance on both flat and complex irregular surfaces without task-specific retraining?
• RQ5 What is the impact of domain-specific fine-tuning on IOU and F-score in thermal image segmentation for defect detection?

Key findings

• DefectSAM achieves 100.0% F-score on both flat-type and R-type thermal images, outperforming all existing methods including MedSAM and 3D-UNet++.
• DefectSAM achieves 78.58% IOU on flat-type images and 81.98% IOU on R-type images, representing a 18.05% and 24.68% improvement over 3D-UNet++-Sequence PCA, respectively.
• The model successfully segments defects with complex, non-circular shapes, avoiding the circular bias observed in MedSAM, which incorrectly segments rectangular defects as circular.
• DefectSAM demonstrates superior generalization, detecting all defects even in low-contrast, high-noise conditions where traditional models fail.
• The model maintains high performance without requiring additional training on thermal data, showing strong zero-shot transferability from natural image pre-training.
• Expert-provided prompts significantly reduce missed detections and improve shape accuracy, especially for small and weak defects.