[Paper Review] Inducing High Energy-Latency of Large Vision-Language Models with Verbose Images

Kuofeng Gao, Yang Bai+5 more|arXiv (Cornell University)|Jan 20, 2024

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes verbose images—imperceptible adversarial perturbations that induce large vision-language models (VLMs) to generate excessively long, complex, and diverse text sequences during inference. By delaying the end-of-sequence (EOS) token, increasing token-level uncertainty, and enhancing sequence-level diversity through three loss objectives, the method increases energy and latency costs by up to 8.56× on ImageNet and 7.87× on MS-COCO across multiple VLMs, demonstrating a novel availability-based evasion attack.

ABSTRACT

Large vision-language models (VLMs) such as GPT-4 have achieved exceptional performance across various multi-modal tasks. However, the deployment of VLMs necessitates substantial energy consumption and computational resources. Once attackers maliciously induce high energy consumption and latency time (energy-latency cost) during inference of VLMs, it will exhaust computational resources. In this paper, we explore this attack surface about availability of VLMs and aim to induce high energy-latency cost during inference of VLMs. We find that high energy-latency cost during inference of VLMs can be manipulated by maximizing the length of generated sequences. To this end, we propose verbose images, with the goal of crafting an imperceptible perturbation to induce VLMs to generate long sentences during inference. Concretely, we design three loss objectives. First, a loss is proposed to delay the occurrence of end-of-sequence (EOS) token, where EOS token is a signal for VLMs to stop generating further tokens. Moreover, an uncertainty loss and a token diversity loss are proposed to increase the uncertainty over each generated token and the diversity among all tokens of the whole generated sequence, respectively, which can break output dependency at token-level and sequence-level. Furthermore, a temporal weight adjustment algorithm is proposed, which can effectively balance these losses. Extensive experiments demonstrate that our verbose images can increase the length of generated sequences by 7.87 times and 8.56 times compared to original images on MS-COCO and ImageNet datasets, which presents potential challenges for various applications. Our code is available at https://github.com/KuofengGao/Verbose_Images.

Motivation & Objective

• To investigate the vulnerability of large vision-language models (VLMs) to adversarial attacks that increase energy and latency during inference.
• To explore whether the length of generated sequences correlates with energy consumption and latency in VLMs.
• To develop a method that induces high energy-latency cost in VLMs through imperceptible image perturbations.
• To design a loss-based optimization framework that delays EOS token generation, increases output uncertainty, and enhances token diversity.

Proposed method

• Proposes a delayed EOS loss to postpone the generation of the end-of-sequence token, thereby extending sequence length.
• Introduces an uncertainty loss to increase randomness in token prediction, breaking token-level output dependency.
• Employs a token diversity loss to promote variation across all tokens in the generated sequence, reducing sequence-level predictability.
• Uses a temporal weight adjustment algorithm to balance the three loss objectives during optimization.
• Applies a gradient-based optimization process with momentum to generate imperceptible perturbations on input images.
• Employs NVML for real-time energy measurement and records inference latency to evaluate attack effectiveness.

Experimental results

Research questions

• RQ1 Is there a strong positive correlation between the length of generated sequences and energy consumption/latency in VLMs?
• RQ2 Can adversarial perturbations on visual inputs be used to induce significantly longer text generations in VLMs?
• RQ3 How can the generation process be manipulated to delay EOS token occurrence without altering semantic content?
• RQ4 To what extent can uncertainty and diversity in token generation contribute to longer sequences?
• RQ5 Can a balanced multi-loss optimization framework effectively induce high energy-latency with imperceptible perturbations?

Key findings

• Energy consumption and latency in VLMs show a nearly linear positive correlation with the length of generated sequences.
• Verbose images increase sequence length by 7.87× on MS-COCO and 8.56× on ImageNet compared to original images across four VLMs.
• The method induces complex, hallucinated, and diverse outputs with dispersed attention maps across visual tokens.
• The temporal weight adjustment algorithm effectively balances the three loss objectives, improving optimization stability and attack success.
• Experiments show that momentum-based optimization significantly improves attack performance, with 0.9 momentum yielding the highest sequence length.
• The attack remains effective across diverse VLMs, including BLIP-2, InstructBLIP, and MiniGPT-4, even with advanced LLM backbones.