[Paper Review] 2nd Place Winning Solution for the CVPR2023 Visual Anomaly and Novelty Detection Challenge: Multimodal Prompting for Data-centric Anomaly Detection

Yunkang Cao, Xiaohao Xu+4 more|arXiv (Cornell University)|Jun 15, 2023

Anomaly Detection Techniques and ApplicationsComputer Science3 citations

TL;DR

This paper proposes SAA+, a data-centric, zero-shot anomaly segmentation framework that leverages multimodal prompts—derived from domain expert knowledge and target image context—to regularize foundation models like SAM and GroundingDINO. By integrating language, property, saliency, and confidence prompts, SAA+ achieves state-of-the-art performance on VisA and MVTec-AD benchmarks, significantly reducing false positives compared to baseline prompt-based methods.

ABSTRACT

This technical report introduces the winning solution of the team Segment Any Anomaly for the CVPR2023 Visual Anomaly and Novelty Detection (VAND) challenge. Going beyond uni-modal prompt, e.g., language prompt, we present a novel framework, i.e., Segment Any Anomaly + (SAA$+$), for zero-shot anomaly segmentation with multi-modal prompts for the regularization of cascaded modern foundation models. Inspired by the great zero-shot generalization ability of foundation models like Segment Anything, we first explore their assembly (SAA) to leverage diverse multi-modal prior knowledge for anomaly localization. Subsequently, we further introduce multimodal prompts (SAA$+$) derived from domain expert knowledge and target image context to enable the non-parameter adaptation of foundation models to anomaly segmentation. The proposed SAA$+$ model achieves state-of-the-art performance on several anomaly segmentation benchmarks, including VisA and MVTec-AD, in the zero-shot setting. We will release the code of our winning solution for the CVPR2023 VAN.

Motivation & Objective

• To address the high false-positive rate in zero-shot anomaly segmentation caused by ambiguous, class-agnostic language prompts.
• To improve zero-shot generalization of foundation models for anomaly detection without fine-tuning.
• To integrate domain-specific expert knowledge and dynamic image context into prompt design for better anomaly localization.
• To develop a data-centric framework that regularizes foundation models via multimodal prompting instead of model retraining.
• To achieve state-of-the-art performance on VisA and MVTec-AD under zero-shot settings.

Proposed method

• Constructs a baseline, Segment Any Anomaly (SAA), by cascading a prompt-guided object detector (GroundingDINO) and a segmentation model (SAM).
• Introduces multimodal prompting in SAA+ using four prompt types: language prompts (e.g., 'defect'), property prompts (e.g., 'overlong'), saliency prompts (from image features), and confidence prompts (from top-K region scores).
• Uses visual saliency maps from a pre-trained WideResNet50 to guide region selection and improve localization accuracy.
• Applies confidence ranking to refine candidate regions by selecting top-K predictions based on score, reducing noise.
• Combines multiple prompt signals via weighted aggregation to produce the final anomaly map using a soft voting mechanism.
• Employs a prompt-regularization strategy that avoids model fine-tuning, enabling full zero-shot adaptation.

Figure 1 : Towards segmenting any anomaly without training, we first construct a vanilla baseline (Segment Any Anomaly, SAA) by prompting into a cascade of anomaly region generator ( e.g. , a prompt-guided object detection foundation model [ 2 ] ) and anomaly region refiner ( e.g. , a segmentation f

Experimental results

Research questions

• RQ1 Can multimodal prompting significantly reduce false positives in zero-shot anomaly segmentation compared to naive language prompts?
• RQ2 How effective are domain expert-derived prompts (e.g., specific defect descriptions) in improving localization accuracy?
• RQ3 To what extent can image context, such as visual saliency and confidence scores, improve the robustness of foundation model predictions?
• RQ4 Does combining multiple prompt modalities lead to better generalization than single-modality prompting?
• RQ5 Can a data-centric, prompt-based approach achieve state-of-the-art performance without any model fine-tuning?

Key findings

• SAA+ achieves a max-F1-pixel score of 27.07 on VisA and 39.40 on MVTec-AD, significantly outperforming prior methods including WinClip, ClipSeg, and UTAD.
• The ablation study shows that removing any of the four prompt types (language, property, saliency, confidence) leads to performance drops, confirming the contribution of each component.
• The removal of language prompts (w/o 𝒫ᴸ) results in a 3.78-point drop on VisA, indicating that even basic prompts are essential for baseline localization.
• The inclusion of saliency prompts (w/o 𝒫ˢ) leads to a 7.68-point drop on MVTec-AD, highlighting the importance of spatial context in anomaly detection.
• The confidence prompt (w/o 𝒫ᶜ) reduces performance the most on VisA (from 27.07 to 26.70), showing that score-based filtering enhances prediction reliability.
• Qualitative results confirm that SAA+ detects diverse anomaly types across objects, including subtle and complex defects, with high precision and minimal false alarms.

Figure 2 : Overview of the proposed Segment Any Anomaly + (SAA+) framework. We adapt foundation models to zero-shot anomaly segmentation via multimodal prompt regularization. In specific, apart from naive class-agnostic language prompts, the regularization comes from both domain expert knowledge, in