[Paper Review] Clover: Towards A Unified Video-Language Alignment and Fusion Model

Jingjia Huang, Yinan Li+4 more|arXiv (Cornell University)|Jul 16, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

Clover proposes a unified video-language pre-training model that enhances cross-modal alignment and fusion through a novel Tri-Modal Alignment (TMA) objective, which correlates video, text, and fused multi-modal representations. By incorporating semantic masking and a pair-wise ranking loss, Clover achieves state-of-the-art performance across multiple retrieval and video question answering tasks without sacrificing efficiency, outperforming task-specific models and simple model combinations.

ABSTRACT

Building a universal Video-Language model for solving various video understanding tasks (\emph{e.g.}, text-video retrieval, video question answering) is an open challenge to the machine learning field. Towards this goal, most recent works build the model by stacking uni-modal and cross-modal feature encoders and train it with pair-wise contrastive pre-text tasks. Though offering attractive generality, the resulted models have to compromise between efficiency and performance. They mostly adopt different architectures to deal with different downstream tasks. We find this is because the pair-wise training cannot well \emph{align} and \emph{fuse} features from different modalities. We then introduce extbf{Clover} extemdash a Correlated Video-Language pre-training method extemdash towards a universal Video-Language model for solving multiple video understanding tasks with neither performance nor efficiency compromise. It improves cross-modal feature alignment and fusion via a novel tri-modal alignment pre-training task. Additionally, we propose to enhance the tri-modal alignment via incorporating learning from semantic masked samples and a new pair-wise ranking loss. Clover establishes new state-of-the-arts on multiple downstream tasks, including three retrieval tasks for both zero-shot and fine-tuning settings, and eight video question answering tasks. Codes and pre-trained models will be released at \url{https://github.com/LeeYN-43/Clover}.

Motivation & Objective

• To address the inefficiency and performance compromise in existing unified video-language models that use separate architectures for retrieval and VQA tasks.
• To improve cross-modal feature alignment and fusion by explicitly correlating uni-modal and multi-modal encoders through a novel pre-training objective.
• To maintain high efficiency for large-scale retrieval while achieving strong performance on downstream video understanding tasks.
• To enhance model generalization and discriminative capability through semantic masking and pair-wise ranking loss during pre-training.

Proposed method

• Introduces Tri-Modal Alignment (TMA), a pre-training task that aligns the multi-modal representation with both the video and text uni-modal representations to boost mutual learning.
• Employs a semantic masking strategy to generate masked samples that preserve key semantic content, improving robustness to partial input.
• Incorporates a pair-wise ranking loss to distinguish between masked and original samples, enhancing the model's sensitivity to semantic differences.
• Uses standard masked language modeling (MLM) as a base objective, compatible with existing pre-training frameworks and adding minimal computational overhead.
• Combines TMA with existing contrastive and masked modeling objectives to jointly optimize alignment and fusion.
• Employs a shared encoder architecture with separate uni-modal encoders for video and text, and a joint multi-modal encoder for fusion.

Experimental results

Research questions

• RQ1 Can a unified video-language model achieve both high efficiency in retrieval and high performance in video question answering without architectural compromise?
• RQ2 Does explicitly correlating the outputs of uni-modal and multi-modal encoders through tri-modal alignment improve cross-modal alignment and fusion?
• RQ3 Can semantic masking and pair-wise ranking loss enhance the model’s ability to capture fine-grained semantic differences in video-text pairs?
• RQ4 How does the proposed Tri-Modal Alignment compare to standard contrastive or masked language modeling pre-training in terms of downstream performance?

Key findings

• Clover establishes new state-of-the-art results on three text-video retrieval benchmarks under both zero-shot and fine-tuning settings.
• On eight video question answering benchmarks, Clover consistently outperforms both task-specific models and simple model combinations (COMB).
• The model with Tri-Modal Alignment achieves higher cosine similarity for positive video-text pairs and larger margin between positive and negative pairs compared to the baseline.
• The inclusion of semantic masking and pair-wise ranking loss leads to measurable performance gains, especially on complex datasets like DiDemo with diverse captions.
• Qualitative results show that Clover generates more accurate descriptions and answers, such as correctly identifying 'paper airplane' or 'Lion' instead of vague terms.
• The ablation study confirms that TMA alone improves performance over standard contrastive pre-training, and the full Clover model with all components achieves the best results.