[Paper Review] Rethinking Visual Dependency in Long-Context Reasoning for Large Vision-Language Models

Yucheng Zhou, Zhi Rao+2 more|arXiv (Cornell University)|Oct 25, 2024

Multimodal Machine Learning Applications4 citations

TL;DR

This paper identifies that large vision-language models (LVLMs) suffer from declining visual dependency in long-context reasoning due to overreliance on textual information. It proposes a training-free context pruning method that removes less critical textual tokens based on attention-weight aggregation, significantly improving visual attention and performance across multiple LVLMs on a newly constructed long-context dataset, with optimal pruning rates increasing with context length.

ABSTRACT

Large Vision-Language Models (LVLMs) excel in cross-model tasks but experience performance declines in long-context reasoning due to overreliance on textual information and reduced visual dependency. In this study, we empirically analyze LVLMs in long-context reasoning, revealing that increased context length leads to a higher dependence on language at the expense of visual dependency. To address this issue, we propose a novel training-free context pruning method that selectively removes less critical textual information. Our approach enhances visual dependency and reduces textual noise, thereby improving LVLM performance in long-context reasoning. We validate our method by constructing a long-context dataset, demonstrating its effectiveness across various LVLMs. Moreover, further analysis confirms the robustness of different token pruning strategies and preliminary explores scaling laws between pruning rates and context length.

Motivation & Objective

• To investigate the root cause of performance degradation in large vision-language models (LVLMs) during long-context reasoning tasks.
• To analyze how visual dependency diminishes as context length increases, particularly due to overreliance on textual priors.
• To develop a training-free method that enhances visual dependency by selectively pruning less critical textual tokens.
• To validate the method across multiple LVLMs using a newly constructed long-context dataset.
• To explore scaling laws between pruning rates and context length for optimal performance.

Proposed method

• The method identifies less critical textual tokens using aggregated attention weights across multiple attention heads in the transformer layers.
• It applies a pruning strategy that removes tokens with low cumulative attention scores, reducing textual noise while preserving visual information.
• The approach is training-free and operates by modifying the input context before inference, without fine-tuning the model.
• A theoretical analysis demonstrates that pruning leads to more concentrated and stable attention distributions, favoring visual input integration.
• The method is evaluated across different pruning layers and rates, with performance analyzed via part-of-speech distribution of remaining tokens.
• A long-context dataset is constructed based on the SVIT benchmark to enable systematic evaluation of reasoning performance under extended contexts.

Experimental results

Research questions

• RQ1 Why does visual dependency in LVLMs decline as context length increases in long-context reasoning tasks?
• RQ2 How does the proportion of the target object in an image affect visual dependency stability in long-context settings?
• RQ3 To what extent do cross-modal interactions shift from early to deep layers in LVLMs during long-context reasoning?
• RQ4 Can a training-free context pruning method improve visual dependency and reasoning performance in LVLMs without fine-tuning?
• RQ5 What is the relationship between optimal pruning rate and context length in long-context reasoning?

Key findings

• Performance in long-context reasoning drops by up to 17% for LLaVA when context length increases, indicating a significant decline in visual dependency.
• The proposed pruning method improves performance across multiple LVLMs, with higher gains observed in longer contexts (1.5k–2.5k tokens), especially at a 0.3 pruning rate.
• Pruning based on attention-weight aggregation preserves critical information more effectively than random pruning, as shown by stable part-of-speech distributions in remaining tokens.
• The method leads to more concentrated and stable attention distributions, which enhances visual input integration and model robustness.
• A clear upward trend is observed between context length and optimal pruning rate, suggesting that longer contexts benefit from more aggressive pruning.
• Case studies confirm that the method maintains high accuracy across all context lengths, while the baseline fails beyond 0.5k tokens.