[Paper Review] How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs

Haoqin Tu, Chenhang Cui+7 more|arXiv (Cornell University)|Nov 27, 2023

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper introduces a comprehensive safety evaluation benchmark for Vision-Language Large Models (VLLMs), assessing out-of-distribution (OOD) generalization and adversarial robustness through two novel VQA datasets—OODCV-VQA and Sketchy-VQA—along with targeted attacks on vision and language components. Key findings reveal that VLLMs excel at interpreting OOD visual inputs but fail under OOD textual perturbations, are vulnerable to vision encoder attacks, and suffer from weakened safety protocols due to vision-language pretraining.

ABSTRACT

This work focuses on the potential of Vision LLMs (VLLMs) in visual reasoning. Different from prior studies, we shift our focus from evaluating standard performance to introducing a comprehensive safety evaluation suite, covering both out-of-distribution (OOD) generalization and adversarial robustness. For the OOD evaluation, we present two novel VQA datasets, each with one variant, designed to test model performance under challenging conditions. In exploring adversarial robustness, we propose a straightforward attack strategy for misleading VLLMs to produce visual-unrelated responses. Moreover, we assess the efficacy of two jailbreaking strategies, targeting either the vision or language component of VLLMs. Our evaluation of 21 diverse models, ranging from open-source VLLMs to GPT-4V, yields interesting observations: 1) Current VLLMs struggle with OOD texts but not images, unless the visual information is limited; and 2) These VLLMs can be easily misled by deceiving vision encoders only, and their vision-language training often compromise safety protocols. We release this safety evaluation suite at https://github.com/UCSC-VLAA/vllm-safety-benchmark.

Motivation & Objective

• To address the lack of systematic safety evaluation in Vision LLMs, particularly regarding OOD generalization and adversarial robustness.
• To develop a comprehensive benchmark that tests VLLM safety across diverse failure modes, including visual and textual perturbations.
• To investigate how vision-language pretraining compromises safety protocols in aligned LLMs.
• To evaluate the effectiveness of jailbreaking attacks targeting either the vision or language modality of VLLMs.
• To identify architectural and training design factors that influence VLLM safety, especially in open-source and closed-source models.

Proposed method

• Proposed two new OOD VQA datasets—OODCV-VQA and Sketchy-VQA—using rare visual concepts and counterfactual descriptions to test OOD generalization.
• Designed a simple yet effective attack strategy targeting the CLIP vision encoder by aligning it with irrelevant textual objects to mislead VLLM outputs.
• Evaluated two jailbreaking strategies: white-box and transfer attacks on both vision and language inputs to assess toxicity instruction elicitation.
• Conducted extensive evaluation on 21 models, including open-source VLLMs and GPT-4V, across multiple OOD scenarios and attack types.
• Analyzed model performance across varying LLM architectures, vision encoders, and fine-tuning configurations to isolate safety-related design factors.
• Released the benchmark suite, including datasets and code, to enable reproducibility and future research.

Experimental results

Research questions

• RQ1 How do VLLMs perform under out-of-distribution visual and textual inputs, particularly when the language input is counterfactually perturbed?
• RQ2 Can simple vision encoder attacks, such as those based on CLIP ViT, effectively mislead VLLMs into generating irrelevant or incorrect responses?
• RQ3 To what extent can jailbreaking attacks be effective by manipulating only the visual input, without altering the language prompt?
• RQ4 Why do vision-language pretraining procedures weaken safety protocols in aligned LLMs, and how does this affect model behavior?
• RQ5 What architectural and training choices—such as model scale, vision encoder type, or data diversity—most significantly impact VLLM safety?

Key findings

• VLLMs demonstrate strong performance on OOD visual inputs (e.g., unusual textures, poses, shapes) but struggle significantly when the textual input is perturbed in a counterfactual manner.
• Models like GPT-4V and open-source VLLMs show a 33.5% lower attack success rate (ASR) on vision encoder attacks compared to other models, indicating stronger resistance when safety mechanisms are active.
• Sketch-based images pose a major challenge: even simple yes/no questions on sketches result in high error rates, with models like GPT-4V failing on basic recognition tasks.
• A simple CLIP ViT-based attack achieves comparable or stronger success in misleading VLLMs than ensemble-based attacks, showing that vision encoder manipulation alone can be highly effective.
• Vision-language pretraining consistently undermines safety protocols in LLMs, with models trained on standard datasets failing to reject toxic or irrelevant responses when exposed to adversarial inputs.
• InstructBLIP outperforms MiniGPT4 despite similar architecture due to its use of 13 diverse vision-language datasets, highlighting the critical role of data diversity and quality in safety and robustness.