[Paper Review] Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs

Qizhe Zhang, Aosong Cheng+7 more|arXiv (Cornell University)|Dec 2, 2024

Retinal Imaging and AnalysisMedicine3 citations

TL;DR

This paper proposes FasterVLM, a training-free visual token pruning method that improves VLM inference efficiency by leveraging [CLS] token attention from the visual encoder instead of relying on inaccurate text-visual cross-attention in LLMs. By pruning less important visual tokens early, FasterVLM achieves 95% token reduction while retaining 90% of LLaVA-1.5-7B's performance, significantly outperforming existing methods at high reduction ratios.

ABSTRACT

Large vision-language models (LVLMs) generally contain significantly more visual tokens than their textual counterparts, resulting in a considerable computational burden. Recent efforts have been made to tackle this issue by pruning visual tokens early within the language model. Most existing works use attention scores between text and visual tokens to assess the importance of visual tokens. However, in this study, we first analyze the text-visual attention in the language model and find that this score is not an ideal indicator for token pruning. Based on the analysis, We propose VisPruner, a plug-and-play method that utilizes visual cues for more effective token pruning in LVLMs. Specifically, we first use visual attention to select a limited number of significant tokens. Then, we remove duplicate tokens from the remaining ones based on their similarity. By retaining diverse tokens alongside the initially selected important tokens, we maximally preserve the visual information of the input image. Experimental results demonstrate that our VisPruner sustains strong performance across various VLM architectures and reduction ratios, significantly outperforming existing methods based on text-visual attention. Notably, without any training, VisPruner can reduce the FLOPs of LLaVA-1.5-7B by 91% and inference latency by 75%, while maintaining comparable performance. Our code is available at https://github.com/Theia-4869/VisPruner.

Motivation & Objective

• To address the inefficiency of current VLMs caused by excessive visual tokens, which increase computational cost during inference.
• To identify the root cause of performance degradation in existing text-visual attention-based pruning methods, particularly at high reduction ratios.
• To develop a more accurate, training-free method for visual token pruning by leveraging attention from the [CLS] token in the visual encoder.
• To ensure compatibility with FlashAttention by pruning tokens before they enter the LLM, maximizing inference speed.
• To demonstrate generalization across diverse VLM architectures, including LLaVA-1.5, LLaVA-NeXT, and Video-LLaVA.

Proposed method

• FasterVLM evaluates visual token importance using attention weights between the [CLS] token and image tokens in the visual encoder, which are more accurate than text-visual attention in LLMs.
• It prunes visual tokens with the lowest [CLS] attention scores before they are processed by the LLM, eliminating redundant computation early in the pipeline.
• The method is training-free and operates entirely within the visual encoder, making it compatible with optimized attention mechanisms like FlashAttention.
• The approach avoids the attention shift and dispersion problems inherent in LLM-based text-visual attention, which misalign token importance with actual task relevance.
• Pruning is performed based on a global attention mechanism in the visual encoder, which provides more concentrated and reliable importance scores.
• The method is applied across multiple VLMs, including LLaVA-1.5, LLaVA-NeXT, and Video-LLaVA, demonstrating broad applicability.

Experimental results

Research questions

• RQ1 Why do existing text-visual attention-based pruning methods degrade significantly at high token reduction ratios?
• RQ2 Can [CLS] token attention in the visual encoder serve as a more accurate and stable indicator of visual token importance than LLM-based cross-attention?
• RQ3 Does pruning visual tokens based on [CLS] attention lead to better performance and efficiency trade-offs in VLM inference?
• RQ4 Can this method be applied universally across different VLM architectures without retraining?
• RQ5 How does the proposed method compare in efficiency and accuracy when integrated with FlashAttention?

Key findings

• FasterVLM achieves 95% visual token reduction while maintaining 90% of the performance of vanilla LLaVA-1.5-7B, significantly outperforming existing methods at high reduction ratios.
• On 10 benchmarks, FasterVLM outperforms other methods on 9 tasks and matches them on the remaining one at a 90% reduction ratio.
• FasterVLM maintains 89.28% average accuracy on LLaVA-1.5-7B at 95% token reduction, compared to only 67.22% for FastV using FlashAttention.
• The method reduces FLOPs by up to 88% and inference time by up to 77% compared to baseline methods, with minimal memory overhead.
• Ablation studies confirm that [CLS] attention is more concentrated and less prone to attention shift than LLM-based text-visual attention.
• FasterVLM enables compatibility with FlashAttention by pruning before the LLM, resulting in faster inference than methods that prune within the LLM.