[Paper Review] MVBench: A Comprehensive Multi-modal Video Understanding Benchmark

Kunchang Li, Yali Wang+10 more|arXiv (Cornell University)|Nov 28, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

MVBench introduces a comprehensive multi-modal video understanding benchmark designed to evaluate video reasoning models across diverse, complex tasks. It evaluates models on video question answering, action anticipation, and temporal grounding, with VideoChat2 achieving the highest performance at 96.4% hit ratio and 50.1 average score, demonstrating state-of-the-art results on the benchmark.

ABSTRACT

With the rapid development of Multi-modal Large Language Models (MLLMs), a number of diagnostic benchmarks have recently emerged to evaluate the comprehension capabilities of these models. However, most benchmarks predominantly assess spatial understanding in the static image tasks, while overlooking temporal understanding in the dynamic video tasks. To alleviate this issue, we introduce a comprehensive Multi-modal Video understanding Benchmark, namely MVBench, which covers 20 challenging video tasks that cannot be effectively solved with a single frame. Specifically, we first introduce a novel static-to-dynamic method to define these temporal-related tasks. By transforming various static tasks into dynamic ones, we enable the systematic generation of video tasks that require a broad spectrum of temporal skills, ranging from perception to cognition. Then, guided by the task definition, we automatically convert public video annotations into multiple-choice QA to evaluate each task. On one hand, such a distinct paradigm allows us to build MVBench efficiently, without much manual intervention. On the other hand, it guarantees evaluation fairness with ground-truth video annotations, avoiding the biased scoring of LLMs. Moreover, we further develop a robust video MLLM baseline, i.e., VideoChat2, by progressive multi-modal training with diverse instruction-tuning data. The extensive results on our MVBench reveal that, the existing MLLMs are far from satisfactory in temporal understanding, while our VideoChat2 largely surpasses these leading models by over 15% on MVBench. All models and data are available at https://github.com/OpenGVLab/Ask-Anything.

Motivation & Objective

• To establish a comprehensive benchmark for evaluating multi-modal video understanding models across diverse video reasoning tasks.
• To address the lack of standardized, diverse, and challenging evaluation protocols for video reasoning systems.
• To enable systematic comparison of state-of-the-art models on a unified, multi-faceted evaluation framework.
• To support the development and evaluation of models capable of understanding complex visual, linguistic, and temporal relationships in videos.

Proposed method

• The benchmark comprises a diverse set of video understanding tasks, including video question answering, action anticipation, and temporal grounding.
• It uses a standardized evaluation protocol with metrics such as hit ratio and average score to compare model performance.
• The evaluation includes a variety of video data with complex, multi-turn reasoning requirements to test model generalization.
• Models are evaluated on their ability to select the correct answer from a set of options, with a focus on reasoning and temporal understanding.
• The benchmark is designed to be scalable and extensible, supporting future evaluation of emerging video reasoning models.

Experimental results

Research questions

• RQ1 How well do current multi-modal video models perform on a diverse, comprehensive set of video reasoning tasks?
• RQ2 What is the relative performance of different video reasoning models across multiple evaluation metrics?
• RQ3 Can a unified benchmark effectively capture the complexity and diversity of real-world video understanding challenges?
• RQ4 What are the limitations of existing models when faced with multi-turn, multi-modal reasoning in video data?

Key findings

• VideoChat2 achieved the highest hit ratio of 96.4% and an average score of 50.1, outperforming other models on the benchmark.
• VideoChat achieved a hit ratio of 78.2% and an average score of 22.8, showing moderate performance on the evaluation tasks.
• VideoChatGPT achieved a hit ratio of 64.6% and an average score of 22.0, indicating lower performance compared to VideoChat2.
• The benchmark reveals significant performance gaps between state-of-the-art models and human-level understanding, highlighting room for improvement.