[Paper Review] CLIP4Caption ++: Multi-CLIP for Video Caption

Mingkang Tang, Zhanyu Wang+3 more|arXiv (Cornell University)|Oct 11, 2021

Multimodal Machine Learning Applications13 references4 citations

TL;DR

CLIP4Caption++ proposes a multi-CLIP video captioning framework that enhances visual feature extraction using three pre-trained CLIP models (ViT-B/32, ViT-B/16, RN50x16) and SlowFast for motion features, integrates video subtitles for semantic enrichment, applies TSN-based data augmentation, and employs word-level and sentence-level ensemble strategies. The method achieves state-of-the-art performance with 86.5 CIDEr on VATEX, 148.4 on YC2C, and 64.5 on TVC, winning first place in the VALUE Challenge 2021 captioning task.

ABSTRACT

This report describes our solution to the VALUE Challenge 2021 in the captioning task. Our solution, named CLIP4Caption++, is built on X-Linear/X-Transformer, which is an advanced model with encoder-decoder architecture. We make the following improvements on the proposed CLIP4Caption++: We employ an advanced encoder-decoder model architecture X-Transformer as our main framework and make the following improvements: 1) we utilize three strong pre-trained CLIP models to extract the text-related appearance visual features. 2) we adopt the TSN sampling strategy for data enhancement. 3) we involve the video subtitle information to provide richer semantic information. 3) we introduce the subtitle information, which fuses with the visual features as guidance. 4) we design word-level and sentence-level ensemble strategies. Our proposed method achieves 86.5, 148.4, 64.5 CIDEr scores on VATEX, YC2C, and TVC datasets, respectively, which shows the superior performance of our proposed CLIP4Caption++ on all three datasets.

Motivation & Objective

• To improve video captioning performance by enhancing visual feature representation using multiple strong pre-trained CLIP models.
• To enrich semantic understanding by incorporating video subtitle information into the captioning model.
• To improve training generalization through TSN-based frame sampling for data augmentation.
• To enhance model robustness and output quality via word-level and sentence-level ensemble strategies.
• To achieve state-of-the-art performance on benchmark video captioning datasets without additional pre-training or data.

Proposed method

• Utilizes three pre-trained CLIP models—CLIP(ViT-B/32), CLIP(ViT-B/16), and CLIP(RN50x16)—to extract multi-scale, text-aligned visual features from video frames.
• Employs SlowFast(8x8) with ResNet-50 to extract motion features from 64-frame clips using separate slow and fast pathways at different frame rates.
• Applies TSN sampling during training to randomly sample one frame per segment across K segments, increasing data diversity and enhancing generalization.
• Integrates video subtitles by tokenizing and embedding them alongside visual features, with a shared embedding layer and token type embeddings to distinguish modalities.
• Fuses visual and textual features via concatenation and token type embeddings before feeding into the X-Linear/X-Transformer encoder-decoder architecture.
• Implements a two-stage training pipeline: first-stage pre-training on enhanced features, followed by second-stage fine-tuning with SCST to directly optimize CIDEr scores.

Experimental results

Research questions

• RQ1 Can combining multiple pre-trained CLIP models improve visual representation quality for video captioning?
• RQ2 How does incorporating video subtitles affect the semantic richness of generated captions?
• RQ3 To what extent does TSN-based sampling enhance model generalization in video captioning?
• RQ4 Can word-level and sentence-level ensemble strategies significantly improve caption quality beyond single-model outputs?
• RQ5 Does end-to-end training with SCST fine-tuning lead to better CIDEr scores compared to standard supervision?

Key findings

• The ensemble model achieved 86.5 CIDEr on the VATEX test set, surpassing the previous state of the art.
• On YC2C, the ensemble model reached 148.4 CIDEr, demonstrating strong performance on a large-scale, diverse video captioning benchmark.
• The method achieved 64.5 CIDEr on TVC, showing robustness across multiple datasets with varying characteristics.
• The ablation study showed that SCST fine-tuning contributed a 24.1% gain in CIDEr over the baseline, highlighting its effectiveness in optimizing for the target metric.
• The addition of subtitle information improved CIDEr by 0.8 points, indicating its value in enriching semantic context.
• The word-level and sentence-level ensemble strategies together contributed a 31.2% CIDEr gain over the baseline, proving their effectiveness in boosting final output quality.