[Paper Review] CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval

Huaishao Luo, Lei Ji+5 more|arXiv (Cornell University)|Apr 18, 2021

Video Analysis and SummarizationComputer Science46 references113 citations

TL;DR

CLIP4Clip transfers CLIP’s image-language pretraining to end-to-end video-text retrieval, explores three similarity calculators, and achieves state-of-the-art results across multiple video-text datasets.

ABSTRACT

Video-text retrieval plays an essential role in multi-modal research and has been widely used in many real-world web applications. The CLIP (Contrastive Language-Image Pre-training), an image-language pre-training model, has demonstrated the power of visual concepts learning from web collected image-text datasets. In this paper, we propose a CLIP4Clip model to transfer the knowledge of the CLIP model to video-language retrieval in an end-to-end manner. Several questions are investigated via empirical studies: 1) Whether image feature is enough for video-text retrieval? 2) How a post-pretraining on a large-scale video-text dataset based on the CLIP affect the performance? 3) What is the practical mechanism to model temporal dependency between video frames? And 4) The Hyper-parameters sensitivity of the model on video-text retrieval task. Extensive experimental results present that the CLIP4Clip model transferred from the CLIP can achieve SOTA results on various video-text retrieval datasets, including MSR-VTT, MSVC, LSMDC, ActivityNet, and DiDeMo. We release our code at https://github.com/ArrowLuo/CLIP4Clip.

Motivation & Objective

• Assess whether single-image features suffice for video-text retrieval.
• Evaluate the impact of post-pretraining CLIP on large-scale video-text data.
• Model temporal dependencies between video frames effectively.
• Examine hyper-parameter sensitivity for video-text retrieval performance.

Proposed method

• Initialize video and text encoders with CLIP (ViT-B/32).
• Propose three similarity calculator types: parameter-free mean pooling, sequential (LSTM or Transformer), and tight Transformer-based interaction.
• Compare 2D vs 3D linear projections for patch embeddings to capture temporal info.
• Post-pretrain CLIP4Clip on a large-scale video-text dataset (Howto100M-380k) using MIL-NCE.
• Train with a symmetric cross-entropy loss s(v,t) for video-to-text and text-to-video directions.
• Use uniform frame sampling (1 frame per second) and conduct end-to-end finetuning of new modules.
• Initialize new parameters for sequential and tight types from CLIP where possible; fine-tune with Adam and cosine learning rate schedule.
• Pretraining and finetuning details include frame length, token length, and GPU setup as described.

Experimental results

Research questions

• RQ1 Is one single image sufficient for encoding video in video-text retrieval?
• RQ2 What is the effect of post-pretraining CLIP on a large-scale video-text dataset for retrieval performance?
• RQ3 What mechanisms best model temporal dependencies across video frames for retrieval?
• RQ4 How do hyperparameters affect end-to-end video-text retrieval performance across datasets?

Key findings

• CLIP4Clip achieves state-of-the-art results on MSR-VTT, MSVC, LSMDC, ActivityNet, and DiDeMo across various splits and settings.
• A single image is not sufficient to encode video effectively for video-text retrieval.
• Post-pretraining on a large-scale video-text dataset improves performance, especially for zero-shot retrieval.
• Mean pooling (parameter-free) is effective on small datasets, while adding temporal modeling (e.g., self-attention) helps on larger datasets.
• Most tight-type (cross-modal interaction) configurations underperform compared to mean-pooling or sequential types when data is limited, but sequential/transformer-based approaches gain with larger data.