[Paper Review] InternVideo: General Video Foundation Models via Generative and Discriminative Learning

Yi Wang, Kunchang Li+15 more|arXiv (Cornell University)|Dec 6, 2022

Human Pose and Action Recognition91 citations

TL;DR

InternVideo builds a general video foundation model by unifying masked video modeling and video-language contrastive learning, achieving state-of-the-art results across 39 datasets and multiple video tasks.

ABSTRACT

The foundation models have recently shown excellent performance on a variety of downstream tasks in computer vision. However, most existing vision foundation models simply focus on image-level pretraining and adpation, which are limited for dynamic and complex video-level understanding tasks. To fill the gap, we present general video foundation models, InternVideo, by taking advantage of both generative and discriminative self-supervised video learning. Specifically, InternVideo efficiently explores masked video modeling and video-language contrastive learning as the pretraining objectives, and selectively coordinates video representations of these two complementary frameworks in a learnable manner to boost various video applications. Without bells and whistles, InternVideo achieves state-of-the-art performance on 39 video datasets from extensive tasks including video action recognition/detection, video-language alignment, and open-world video applications. Especially, our methods can obtain 91.1% and 77.2% top-1 accuracy on the challenging Kinetics-400 and Something-Something V2 benchmarks, respectively. All of these results effectively show the generality of our InternVideo for video understanding. The code will be released at https://github.com/OpenGVLab/InternVideo .

Motivation & Objective

• Motivate the need for general video foundation models beyond image-focused pretraining.
• Develop a unified representation learning paradigm that combines masked video modeling and multimodal contrastive learning.
• Enable efficient cross-representation interaction to fuse generative and discriminative learned features.
• Validate the approach on a broad benchmark spanning action understanding, video-language alignment, and open-world applications.

Proposed method

• Use VideoMAE-style masked video modeling with high masking ratios to learn spatiotemporal representations.
• Employ video-language contrastive learning based on a CLIP-like framework with a caption decoder for cross-modal fusion.
• Post-train with supervised action recognition (Kinetics-710) to enhance downstream performance.
• Introduce cross-model attention (CMA) to align and fuse masked-video and multimodal representations after separate pretraining.
• Keep backbones frozen during CMA to maintain stability while learning a joint representation.

Experimental results

Research questions

• RQ1 Can a unified representation learned from both masked video modeling and multimodal contrastive learning generalize across diverse video tasks?
• RQ2 Does cross-model attention effectively fuse generative and discriminative video features without joint end-to-end training?
• RQ3 How scalable are the components (VideoMAE, UniFormerV2-based multimodal encoder) when scaled to large datasets and model sizes?
• RQ4 What is the impact of supervised post-pretraining on downstream performance across action understanding and video-language tasks?
• RQ5 How does InternVideo perform on open-world and zero-shot settings compared to task-specific models?

Key findings

• Achieves state-of-the-art results on 39 datasets across action understanding, video-language alignment, and open-world tasks.
• On Kinetics-400, InternVideo-D reaches 90.9% top-1 accuracy and InternVideo-T reaches 91.1% (with a +1.2 boost over base for K400).
• On Something-Something V1/V2, ActivityNet, HACS, and HMDB51, InternVideo delivers substantial improvements over prior SOTA methods (e.g., SthSthV1: 70.0%, SthSthV2: 77.2%, ActivityNet: 94.3%, HACS: 95.5%, HMDB51: 89.3%).
• Temporal and spatiotemporal localization tasks (THUMOS-14, ActivityNet-v1.3, HACS, FineAction) show InternVideo providing competitive to state-of-the-art mAP improvements (e.g., THUMOS-14 71.58 with ActionFormer head).
• Video-language tasks show strong retrieval and QA performance; e.g., MSR-VTT/Videos-to-Text retrieval and Video QA gains over baselines; zero-shot and open-set transfers are robust.
• Cross-model interaction (CMA) enables effective fusion of MAE and multimodal features while keeping backbones frozen, offering a computationally tractable training recipe.