[Paper Review] Myriad: Large Multimodal Model by Applying Vision Experts for Industrial Anomaly Detection

Yuanze Li, Haolin Wang+7 more|arXiv (Cornell University)|Oct 29, 2023

vaccines and immunoinformatics approaches9 citations

TL;DR

Myriad integrates vision-expert anomaly maps into a MiniGPT-4 based large multimodal model to achieve definite industrial anomaly detection with detailed descriptions, surpassing several state-of-the-art IAD methods in 1-class and few-shot settings.

ABSTRACT

Due to the training configuration, traditional industrial anomaly detection (IAD) methods have to train a specific model for each deployment scenario, which is insufficient to meet the requirements of modern design and manufacturing. On the contrary, large multimodal models~(LMMs) have shown eminent generalization ability on various vision tasks, and their perception and comprehension capabilities imply the potential of applying LMMs on IAD tasks. However, we observe that even though the LMMs have abundant knowledge about industrial anomaly detection in the textual domain, the LMMs are unable to leverage the knowledge due to the modality gap between textual and visual domains. To stimulate the relevant knowledge in LMMs and adapt the LMMs towards anomaly detection tasks, we introduce existing IAD methods as vision experts and present a novel large multimodal model applying vision experts for industrial anomaly detection~(abbreviated to {Myriad}). Specifically, we utilize the anomaly map generated by the vision experts as guidance for LMMs, such that the vision model is guided to pay more attention to anomalous regions. Then, the visual features are modulated via an adapter to fit the anomaly detection tasks, which are fed into the language model together with the vision expert guidance and human instructions to generate the final outputs. Extensive experiments are applied on MVTec-AD, VisA, and PCB Bank benchmarks demonstrate that our proposed method not only performs favorably against state-of-the-art methods, but also inherits the flexibility and instruction-following ability of LMMs in the field of IAD. Source code and pre-trained models are publicly available at \url{https://github.com/tzjtatata/Myriad}.

Motivation & Objective

• Motivate industrial anomaly detection (IAD) needs beyond binary anomaly maps and scores, aiming for definite judgments and detailed descriptions.
• Leverage existing IAD models as vision experts to inject domain knowledge into a large multimodal model.
• Create a tokenization and guidance mechanism so LLMs can utilize expert priors for IAD tasks.
• Enhance visual representations with a domain adapter to bridge gaps between generic and industrial imagery.
• Demonstrate improved performance on standard IAD benchmarks and provide qualitative anomaly descriptions.

Proposed method

• Use MiniGPT-4 as the base LMM and introduce an Expert Perception module to convert vision-expert outputs into tokens readable by the LLM (VE-Tokenizer).
• Attach a domain adapter to the vision encoder to improve industrial visual feature extraction for anomaly detection.
• Develop a Vision Expert Instructor to produce expert query tokens that interact with visual features via cross-attention in Q-Former, enabling vision-language alignment guided by the expert prior.
• Propose Expert-Driven Vision-Language Extraction to fuse domain-specific vision-language representations with the LLM, mitigating vision-expert errors.
• Train with an IAD instruction dataset comprising templates and synthetic anomalies to enable instruction-following for anomaly description tasks.

Experimental results

Research questions

• RQ1 Can a large multimodal model (LMM) augmented with vision-expert priors provide not only anomaly localization but also definite defect judgments and rich descriptions in IAD?
• RQ2 How can anomaly maps from pre-trained IAD models be converted into tokens understandable by an LLM, and how can cross-attention with domain-adapted visuals improve IAD descriptions?
• RQ3 Do vision-expert priors improve IAD performance in zero-shot, few-shot, and 1-class settings on standard benchmarks?
• RQ4 What is the impact of the VE-Tokenizer and Vision Expert Instructor components on detection accuracy and descriptive capability?
• RQ5 Is there robust performance across MVTec-AD and VisA when incorporating vision experts under various settings?

Key findings

• Myriad achieves competitive 1-class anomaly detection accuracy on MVTec-AD, outperforming several baselines in the reported settings.
• In zero-shot and few-shot settings, Myriad leveraging vision-expert priors demonstrates strong accuracy and localization abilities compared to contemporary methods.
• Qualitative results show Myriad can provide definite anomaly judgments (existence, category, count) and detailed content descriptions (e.g., shadow cast, cut, crack) across datasets.
• Ablation studies confirm the importance of VE-Tokenizer and Vision Expert Instructor for aligning expert priors with industrial vision-language representations, boosting performance.
• Experiments on MVTec-AD and VisA indicate Myriad can function with zero-shot, one-shot, and 1-class configurations, often surpassing or matching state-of-the-art methods in these regimes.