[Paper Review] Dual-Image Enhanced CLIP for Zero-Shot Anomaly Detection

Zhaoxiang Zhang, Hanqiu Deng+2 more|arXiv (Cornell University)|May 8, 2024

Radiation Detection and Scintillator Technologies4 citations

TL;DR

This paper proposes Dual-Image Enhanced CLIP, a zero-shot anomaly detection framework that improves anomaly classification and localization by jointly leveraging two unpaired test images as visual references for each other, while incorporating a test-time adaptation (TTA) module with pseudo-anomaly synthesis. The method achieves state-of-the-art performance on MVTecAD and VisA benchmarks without additional training, demonstrating the effectiveness of visual-linguistic joint scoring and visual reference enrichment.

ABSTRACT

Image Anomaly Detection has been a challenging task in Computer Vision field. The advent of Vision-Language models, particularly the rise of CLIP-based frameworks, has opened new avenues for zero-shot anomaly detection. Recent studies have explored the use of CLIP by aligning images with normal and prompt descriptions. However, the exclusive dependence on textual guidance often falls short, highlighting the critical importance of additional visual references. In this work, we introduce a Dual-Image Enhanced CLIP approach, leveraging a joint vision-language scoring system. Our methods process pairs of images, utilizing each as a visual reference for the other, thereby enriching the inference process with visual context. This dual-image strategy markedly enhanced both anomaly classification and localization performances. Furthermore, we have strengthened our model with a test-time adaptation module that incorporates synthesized anomalies to refine localization capabilities. Our approach significantly exploits the potential of vision-language joint anomaly detection and demonstrates comparable performance with current SOTA methods across various datasets.

Motivation & Objective

• To address the limitations of text-only guidance in CLIP-based zero-shot anomaly detection, which often fails to capture subtle visual anomalies.
• To enhance anomaly detection performance by introducing a dual-image input strategy that uses each image as a visual reference for the other during inference.
• To improve localization accuracy through a test-time adaptation (TTA) module that synthesizes pseudo-anomalies to refine visual representations.
• To achieve strong zero-shot performance in an inductive learning setting, avoiding reliance on test-set distribution knowledge.
• To explore the synergy between visual and textual features in vision-language models for fine-grained anomaly detection.

Proposed method

• The framework processes pairs of unlabelled test images, using each as a visual reference for the other to enrich the feature space during anomaly scoring.
• Anomalies are detected via a joint vision-language scoring mechanism that combines textual prompts with cross-image visual similarity.
• A test-time adaptation (TTA) module is introduced, which applies pseudo-anomaly synthesis (inspired by DRAEM) to refine visual features during inference.
• The TTA module operates with a training step of 2, optimizing for AUROC and PRO scores on anomaly localization and classification.
• The method does not require fine-tuning; all improvements stem from inference-time processing and visual reference fusion.
• The model uses CLIP’s vision and text encoders to compute contrastive embeddings, with anomaly scores derived from cross-modality similarity.

Experimental results

Research questions

• RQ1 Can the use of dual unpaired images as mutual visual references improve zero-shot anomaly detection beyond text-only prompting?
• RQ2 How does incorporating visual references from another image affect the model's ability to detect and localize subtle anomalies?
• RQ3 To what extent can test-time adaptation with pseudo-anomaly synthesis enhance the localization performance of a pretrained CLIP model?
• RQ4 Does the joint use of visual and textual information lead to more robust anomaly detection than either modality alone?
• RQ5 Can a framework that avoids fine-tuning still achieve SOTA performance in zero-shot inductive anomaly detection?

Key findings

• The dual-image input strategy significantly improves both anomaly classification (AUROC) and localization (PRO), with AUROC reaching 93.2% on MVTecAD when combined with TTA.
• The TTA module with pseudo-anomaly synthesis increases F1Max to 42.4 and PRO to 84.0, demonstrating improved localization balance.
• Optimal performance is achieved with a TTA training step of 2, as higher steps lead to performance degradation.
• The method achieves state-of-the-art performance on MVTecAD and VisA in the zero-shot inductive learning setting, outperforming several SOTA baselines.
• Even when reference images contain anomalies, they still serve as effective references, indicating robustness to visual noise in the reference pair.
• The joint visual-language scoring mechanism enhances detection accuracy, particularly for complex anomalies like missing cables or misplaced objects.