[Paper Review] Understanding Chinese Video and Language via Contrastive Multimodal Pre-Training

Chenyi Lei, Shixian Luo+7 more|arXiv (Cornell University)|Apr 19, 2021

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes Victor, a contrastive multimodal pre-training framework for Chinese video-language understanding that introduces novel reconstructive and contrastive proxy tasks to capture complex spatiotemporal and sequential relationships. Trained on the large-scale AlivOL-10M dataset, Victor achieves state-of-the-art performance across multiple downstream tasks, including video retrieval, classification, recommendation, and captioning.

ABSTRACT

The pre-trained neural models have recently achieved impressive performances in understanding multimodal content. However, it is still very challenging to pre-train neural models for video and language understanding, especially for Chinese video-language data, due to the following reasons. Firstly, existing video-language pre-training algorithms mainly focus on the co-occurrence of words and video frames, but ignore other valuable semantic and structure information of video-language content, e.g., sequential order and spatiotemporal relationships. Secondly, there exist conflicts between video sentence alignment and other proxy tasks. Thirdly, there is a lack of large-scale and high-quality Chinese video-language datasets (e.g., including 10 million unique videos), which are the fundamental success conditions for pre-training techniques. In this work, we propose a novel video-language understanding framework named VICTOR, which stands for VIdeo-language understanding via Contrastive mulTimOdal pRe-training. Besides general proxy tasks such as masked language modeling, VICTOR constructs several novel proxy tasks under the contrastive learning paradigm, making the model be more robust and able to capture more complex multimodal semantic and structural relationships from different perspectives. VICTOR is trained on a large-scale Chinese video-language dataset, including over 10 million complete videos with corresponding high-quality textual descriptions. We apply the pre-trained VICTOR model to a series of downstream applications and demonstrate its superior performances, comparing against the state-of-the-art pre-training methods such as VideoBERT and UniVL. The codes and trained checkpoints will be publicly available to nourish further developments of the research community.

Motivation & Objective

• Address the lack of large-scale, high-quality Chinese video-language datasets for pre-training.
• Overcome limitations in existing video-language pre-training models that fail to exploit spatiotemporal and sequential relationships in video and text.
• Resolve conflicts between video-sentence alignment and other proxy tasks by introducing contrastive learning.
• Develop a robust pre-training framework that enhances multimodal representation learning for downstream video-language tasks in Chinese.
• Demonstrate the effectiveness of the proposed model on diverse downstream applications, including retrieval, classification, recommendation, and captioning.

Proposed method

• Propose Victor, an encoder-decoder Transformer-based framework for video-language pre-training.
• Introduce novel reconstructive proxy tasks: Masked Frame Order Modeling (MFOM) and Masked Sentence Order Modeling (MSOM) to capture sequential structure.
• Design contrastive proxy tasks, including dual Video-Text Alignment (dual-VSA), to improve robustness and reduce interference from mismatched pairs.
• Leverage a large-scale Chinese video-language dataset, AlivOL-10M, containing over 10 million videos with high-quality textual descriptions.
• Train the model using a combination of reconstructive and contrastive objectives to jointly optimize multimodal representation learning.
• Fine-tune Victor on downstream tasks using [CLS] token embeddings for discriminative tasks and decoder-based generation for captioning.

Figure 1. General proxy tasks that are widely used in existing video-language pre-training methods.

Experimental results

Research questions

• RQ1 Can contrastive learning improve multimodal representation learning in video-language pre-training by reducing interference from mismatched video-text pairs?
• RQ2 To what extent do novel proxy tasks like MFOM and MSOM enhance the model's ability to capture sequential and structural relationships in video and language?
• RQ3 How does pre-training on a large-scale, high-quality Chinese video-language dataset impact downstream performance compared to existing methods?
• RQ4 Can the proposed framework outperform state-of-the-art models like VideoBERT and UniVL across diverse video-language tasks?
• RQ5 What is the contribution of modeling spatiotemporal relationships (via Intra- and Inter-MFM) to classification and retrieval tasks?

Key findings

• Victor achieves the best performance on video retrieval, with R@10 of 86.7%, outperforming all baselines including UniVL and VideoBERT.
• On multi-level video classification, Victor achieves 92.1% Top-1 accuracy on the LeafCate task and 89.3% on the TopCate task, surpassing M3 and M4.
• In video recommendation, Victor achieves an AUC of 0.892 and HR@5 of 0.721, significantly outperforming the DIN baseline and other pre-trained models.
• For multimodal video captioning, Victor achieves BLEU-4 of 38.2, Meteor of 28.9, and Cider of 124.1, exceeding UniVL’s performance on all metrics.
• Ablation studies confirm that MFOM and MSOM significantly improve sequential reasoning, while dual-VSA and MFM enhance retrieval and classification accuracy.
• Training on the full AlivOL-10M dataset (10M) yields significantly better results than on a 100K subset, validating the importance of large-scale data.

Figure 2. An example of the videos in Alivol -10M dataset. Besides the high-resolution video frames and human-created title, the video also contains a long-text abstract and a related e-commerce product with images. Each video has three types of categories: plot category, coarse-grained product cate