[Paper Review] MLVU: Benchmarking Multi-task Long Video Understanding

Junjie Zhou, Yan Shu+10 more|arXiv (Cornell University)|Jun 6, 2024

Human Pose and Action Recognition4 citations

TL;DR

This paper introduces MLVU, a comprehensive benchmark for multi-task long video understanding that addresses limitations in existing datasets by featuring long videos (3–120 minutes), diverse video genres (movies, surveillance, egocentric, cartoons, game videos), and nine varied evaluation tasks. Empirical evaluation of 20 MLLMs reveals significant performance degradation with increasing video length, with even GPT-4o achieving only 64.6% average accuracy on multiple-choice tasks, highlighting major challenges in long-context reasoning and global video comprehension.

ABSTRACT

The evaluation of Long Video Understanding (LVU) performance poses an important but challenging research problem. Despite previous efforts, the existing video understanding benchmarks are severely constrained by several issues, especially the insufficient lengths of videos, a lack of diversity in video types and evaluation tasks, and the inappropriateness for evaluating LVU performances. To address the above problems, we propose a new benchmark called MLVU (Multi-task Long Video Understanding Benchmark) for the comprehensive and in-depth evaluation of LVU. MLVU presents the following critical values: extit{1)} The substantial and flexible extension of video lengths, which enables the benchmark to evaluate LVU performance across a wide range of durations. extit{2)} The inclusion of various video genres, e.g., movies, surveillance footage, egocentric videos, cartoons, game videos, etc., which reflects the models' LVU performances in different scenarios. extit{3)} The development of diversified evaluation tasks, which enables a comprehensive examination of MLLMs' key abilities in long-video understanding. The empirical study with 23 latest MLLMs reveals significant room for improvement in today's technique, as all existing methods struggle with most of the evaluation tasks and exhibit severe performance degradation when handling longer videos. Additionally, it suggests that factors such as context length, image-understanding ability, and the choice of LLM backbone can play critical roles in future advancements. We anticipate that MLVU will advance the research of long video understanding by providing a comprehensive and in-depth analysis of MLLMs.

Motivation & Objective

• To address the lack of comprehensive benchmarks for long video understanding (LVU) that support diverse video lengths, genres, and evaluation tasks.
• To overcome limitations in existing benchmarks, which often use short videos (a few seconds), focus on single video types, or include trivial tasks that do not require long-context reasoning.
• To enable a unified, in-depth evaluation of multi-modal LLMs (MLLMs) across multiple understanding capabilities such as reasoning, summarization, and action recognition in long videos.
• To identify key factors affecting MLLM performance in LVU, including context length, image understanding quality, and LLM backbone choice.
• To provide a publicly available, extensible benchmark to accelerate research in long video understanding.

Proposed method

• The benchmark is constructed using long-form videos ranging from 3 minutes to 2 hours, with an average duration of 12 minutes, to enable evaluation across diverse temporal extents.
• Videos are segmented into multiple clips (e.g., first 3, 6, and full duration) to allow task evaluation at different temporal spans.
• The dataset includes nine diverse evaluation tasks—such as multi-choice question answering, free-form captioning, action counting, and temporal ordering—covering reasoning, recognition, and summarization.
• Tasks are designed to require either local clip-level information or global video-level context, ensuring comprehensive assessment of MLLM capabilities.
• Annotations are created with careful curation to minimize copyright risks, including resolution reduction, aspect ratio changes, and frame sparsification if needed.
• The benchmark is publicly released on GitHub and Hugging Face under CC-BY-NC-SA 4.0, with support for future updates and contributions.

Experimental results

Research questions

• RQ1 How do current MLLMs perform on long video understanding tasks when video length exceeds 10 minutes?
• RQ2 To what extent do MLLMs struggle with tasks requiring global video context, such as action counting or temporal ordering?
• RQ3 How does the choice of LLM backbone and model context length affect performance on long video understanding tasks?
• RQ4 What is the performance gap between proprietary and open-source MLLMs in long video understanding?
• RQ5 How do different video genres (e.g., surveillance, egocentric, cartoons) impact MLLM performance and generalization?

Key findings

• GPT-4o achieved the highest performance among 20 evaluated MLLMs, with an average accuracy of 64.6% on multi-choice tasks, indicating significant room for improvement.
• All MLLMs experienced severe performance degradation as video length increased, particularly on tasks requiring fine-grained, long-range reasoning.
• Tasks such as action counting, temporal ordering, and video summarization were consistently challenging for all models, suggesting limitations in long-context comprehension.
• Open-source MLLMs, despite strong performance on single-image tasks, significantly underperformed compared to proprietary models like GPT-4o on long video understanding.
• Context length and image understanding quality were identified as critical factors influencing MLLM performance, with poor image encoders leading to notable performance drops.
• The benchmark revealed that many existing tasks are not suitable for evaluating LVU, as they can be solved using only single-frame or textual cues, not long-range video understanding.