[Paper Review] Foundation Models for Video Understanding: A Survey

Neelu Madan, Andreas Moegelmose+3 more|arXiv (Cornell University)|May 6, 2024

Human Pose and Action Recognition4 citations

TL;DR

This survey presents the first comprehensive analysis of Video Foundation Models (ViFMs), categorizing them into image-based, video-based, and universal foundational models (UFMs). It evaluates over 200 ViFMs across 14 video tasks, revealing that image-based models often outperform video-specific ones and UFMs with multimodal pretraining achieve superior performance, offering a roadmap for future research in video understanding.

ABSTRACT

Video Foundation Models (ViFMs) aim to learn a general-purpose representation for various video understanding tasks. Leveraging large-scale datasets and powerful models, ViFMs achieve this by capturing robust and generic features from video data. This survey analyzes over 200 video foundational models, offering a comprehensive overview of benchmarks and evaluation metrics across 14 distinct video tasks categorized into 3 main categories. Additionally, we offer an in-depth performance analysis of these models for the 6 most common video tasks. We categorize ViFMs into three categories: 1) Image-based ViFMs, which adapt existing image models for video tasks, 2) Video-Based ViFMs, which utilize video-specific encoding methods, and 3) Universal Foundational Models (UFMs), which combine multiple modalities (image, video, audio, and text etc.) within a single framework. By comparing the performance of various ViFMs on different tasks, this survey offers valuable insights into their strengths and weaknesses, guiding future advancements in video understanding. Our analysis surprisingly reveals that image-based foundation models consistently outperform video-based models on most video understanding tasks. Additionally, UFMs, which leverage diverse modalities, demonstrate superior performance on video tasks. We share the comprehensive list of ViFMs studied in this work at: \url{https://github.com/NeeluMadan/ViFM_Survey.git}

Motivation & Objective

• To provide the first comprehensive survey of Video Foundation Models (ViFMs) across diverse video understanding tasks.
• To categorize ViFMs into three types: image-based, video-based, and universal foundational models (UFMs) incorporating multiple modalities.
• To analyze benchmarks, evaluation metrics, and performance across 14 video tasks grouped into three temporal categories.
• To compare ViFM performance across tasks and identify key strengths, weaknesses, and research gaps.
• To guide future research by highlighting underexplored directions such as causal reasoning, viewpoint invariance, domain adaptation, and efficiency.

Proposed method

• Systematically categorize over 200 ViFMs into three groups: image-based (fine-tuned from image models), video-based (trained on video data), and UFMs (multimodal pretraining on image, video, audio, and text).
• Classify 14 video understanding tasks into three temporal categories: action recognition, action localization, and long-form video understanding.
• Compile and analyze a comprehensive list of datasets and evaluation metrics for each task category.
• Conduct a detailed performance comparison of ViFMs across the six most common video tasks, using standardized benchmarks.
• Use qualitative and quantitative analysis to evaluate model behavior, generalization, and robustness across different pretraining strategies and modalities.
• Identify emerging trends and challenges, including memory constraints, viewpoint sensitivity, domain shifts, and computational inefficiency.

Figure 1 . Overview of recent research trends in video understanding. The left bar chart shows a significant increase in publications on this topic, based on data from prestigious conferences and journals. The figure presents statistics showcasing research focusing on generative, discriminative, and

Experimental results

Research questions

• RQ1 How do image-based, video-based, and universal foundational models compare in performance across standard video understanding tasks?
• RQ2 Why do image-based foundation models consistently outperform video-specific models on most video tasks?
• RQ3 To what extent do multimodal (e.g., vision-language-audio-text) universal foundational models improve performance on video understanding tasks?
• RQ4 What are the key limitations of current ViFMs in handling long-form video, viewpoint variations, domain shifts, and edge deployment?
• RQ5 How can causal reasoning and viewpoint-invariant representations enhance the robustness and generalization of ViFMs?

Key findings

• Image-based foundation models consistently outperform video-based models on most video understanding tasks, despite being pre-trained on static images.
• Universal Foundation Models (UFMs), which integrate multiple modalities including text, audio, and video, demonstrate superior performance across a wide range of video tasks.
• The survey identifies a significant research gap in domain adaptation for ViFMs, with current models showing limited robustness to domain shifts such as changes in lighting or location.
• Causal reasoning is identified as a promising future direction to improve long-form video understanding by enabling models to answer 'why', 'what next', and 'what if' questions.
• Viewpoint invariance remains a challenge, particularly for egocentric or birds-eye view videos, suggesting a need for 3D-aware or view-consistent representation learning.
• Efficiency remains a critical bottleneck, with ViFMs often requiring billions of parameters and high computational resources, limiting deployment on edge devices.

Figure 2 . Figure contrasts classical (separate feature extraction, model training) and deep learning (unified framework) approaches in computer vision. It also shows the progression of deep learning approaches for both image and video processing over time (best viewed in color).