[Paper Review] PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning

Lin Xu, Yilin Zhao+4 more|arXiv (Cornell University)|Apr 25, 2024

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes PLLaVA, a parameter-free method to extend image-based LLaVA to video dense captioning by applying temporal feature pooling to mitigate performance saturation from high-norm visual features. It achieves state-of-the-art results on Video ChatGPT (3.48/5, +9% over GPT4V) and MVBench (58.1% accuracy, +14.5% over GPT4V), demonstrating improved robustness and scalability without additional parameters or fine-tuning.

ABSTRACT

Vision-language pre-training has significantly elevated performance across a wide range of image-language applications. Yet, the pre-training process for video-related tasks demands exceptionally large computational and data resources, which hinders the progress of video-language models. This paper investigates a straight-forward, highly efficient, and resource-light approach to adapting an existing image-language pre-trained model for dense video understanding. Our preliminary experiments reveal that directly fine-tuning pre-trained image-language models with multiple frames as inputs on video datasets leads to performance saturation or even a drop. Our further investigation reveals that it is largely attributed to the bias of learned high-norm visual features. Motivated by this finding, we propose a simple but effective pooling strategy to smooth the feature distribution along the temporal dimension and thus reduce the dominant impacts from the extreme features. The new model is termed Pooling LLaVA, or PLLaVA in short. PLLaVA achieves new state-of-the-art performance on modern benchmark datasets for both video question-answer and captioning tasks. Notably, on the recent popular VideoChatGPT benchmark, PLLaVA achieves a score of 3.48 out of 5 on average of five evaluated dimensions, exceeding the previous SOTA results from GPT4V (IG-VLM) by 9%. On the latest multi-choice benchmark MVBench, PLLaVA achieves 58.1% accuracy on average across 20 sub-tasks, 14.5% higher than GPT4V (IG-VLM). Code is available at https://pllava.github.io/

Motivation & Objective

• To address performance saturation and collapse in fine-tuning image-based LLaVA for video tasks.
• To reduce the computational and data cost of training video-language models while maintaining high performance.
• To identify and mitigate the negative impact of high-norm visual features on caption quality and model robustness.
• To enable effective model scaling and post-training optimization for video understanding without retraining.
• To improve video captioning detail and accuracy, especially in motion and object description.

Proposed method

• Proposes a simple temporal pooling strategy to smooth visual feature distributions across frames and reduce dominance of high-norm tokens.
• Applies pooling along the temporal dimension to preserve temporal context while minimizing feature distortion.
• Introduces post-training optimization via weighted fusion of image-based and video-fine-tuned LoRA weights to enhance performance on low-quality video data.
• Uses a learnable fusion ratio (alpha) to balance image and video fine-tuned LoRA weights, optimizing for different benchmarks.
• Employs a grid-based frame encoding strategy with pooling to reduce memory cost while preserving spatial and temporal information.
• Leverages the inherent sequence modeling capability of LLMs to understand temporal dynamics without explicit temporal aggregation.

Experimental results

Research questions

• RQ1 Why does fine-tuning image-based LLaVA on video data lead to performance saturation or collapse despite increased data and model size?
• RQ2 How does the dominance of high-norm visual features affect caption length and quality in video understanding tasks?
• RQ3 Can temporal pooling improve feature distribution and model robustness without adding parameters or retraining?
• RQ4 What is the optimal fusion strategy between image and video fine-tuned LoRA weights for improved performance on diverse video benchmarks?
• RQ5 Does a parameter-free adaptation method outperform existing video-specific pre-training or fine-tuning approaches?

Key findings

• PLLaVA achieves a 3.48/5 average score on the Video ChatGPT benchmark, representing a 9% improvement over GPT4V (IG-VLM).
• On the MVBench multi-choice benchmark, PLLaVA achieves 58.1% average accuracy, 14.5% higher than GPT4V (IG-VLM).
• Temporal pooling reduces feature distortion compared to spatial pooling, as shown by higher similarity between spatial neighbors in token embedding visualization.
• Post-training optimization with a fusion ratio of alpha ≈ 20 yields peak performance on MVBench, while alpha ≈ 4 is optimal for VCG score, indicating task-dependent weight balance.
• The model generates longer, more detailed captions than baselines, correctly identifying actions like badminton instead of volleyball, indicating improved motion and object understanding.
• PLLaVA maintains strong performance across model scaling, showing better scalability than existing methods without performance degradation.