[Paper Review] UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation.

Huaishao Luo, Lei Ji+6 more|arXiv (Cornell University)|Feb 15, 2020

Multimodal Machine Learning Applications62 references93 citations

TL;DR

UniVL proposes a unified video-and-language pre-training model that jointly optimizes for both multimodal understanding and generation using a dual-encoder, cross-encoder, and decoder architecture with five pre-training objectives. Trained on HowTo100M, it achieves state-of-the-art performance across five downstream video-text tasks, demonstrating reduced pretrain-finetune discrepancy for generation tasks.

ABSTRACT

With the recent success of the pre-training technique for NLP and image-linguistic tasks, some video-linguistic pre-training works are gradually developed to improve video-text related downstream tasks. However, most of the existing multimodal models are pre-trained for understanding tasks, leading to a pretrain-finetune discrepancy for generation tasks. This paper proposes UniVL: a Unified Video and Language pre-training model for both multimodal understanding and generation. It comprises four components, including two single-modal encoders, a cross encoder, and a decoder with the Transformer backbone. Five objectives, including video-text joint, conditioned masked language model (CMLM), conditioned masked frame model (CMFM), video-text alignment, and language reconstruction, are designed to train each of the components. We further develop two pre-training strategies, stage by stage pre-training (StagedP) and enhanced video representation (EnhancedV), to make the training process of the UniVL more effective. The pre-train is carried out on a sizeable instructional video dataset HowTo100M. Experimental results demonstrate that the UniVL can learn strong video-text representation and achieves state-of-the-art results on five downstream tasks.

Motivation & Objective

• To address the pretrain-finetune discrepancy in multimodal models that are primarily optimized for understanding but underperform on generation tasks.
• To develop a unified architecture capable of jointly learning video and language representations for both understanding and generation.
• To design a set of pre-training objectives that effectively align video and text modalities while supporting conditional generation.
• To improve training efficiency and representation quality through stage-by-stage pre-training and enhanced video representation strategies.
• To evaluate the model on diverse downstream tasks to validate its effectiveness in both understanding and generation scenarios.

Proposed method

• UniVL employs a Transformer-based architecture with two single-modal encoders for video and text, a cross-encoder for cross-modality interaction, and a decoder for autoregressive generation.
• Five pre-training objectives are used: video-text joint pre-training, conditioned masked language model (CMLM), conditioned masked frame model (CMFM), video-text alignment, and language reconstruction.
• The model is trained using two strategies: stage-by-stage pre-training (StagedP) to progressively refine representations, and enhanced video representation (EnhancedV) to improve video encoding quality.
• Pre-training is conducted on the HowTo100M dataset, a large-scale instructional video collection with paired video and text descriptions.
• The cross-encoder and decoder components are jointly optimized to enable end-to-end learning of multimodal dependencies for generation.
• The architecture supports both encoder-decoder and encoder-only inference modes, enabling flexibility across understanding and generation tasks.

Experimental results

Research questions

• RQ1 Can a unified pre-training framework effectively support both multimodal understanding and generation tasks?
• RQ2 How do conditioned masked modeling objectives (CMLM and CMFM) improve cross-modal representation learning?
• RQ3 To what extent does stage-by-stage pre-training and enhanced video representation improve model performance?
• RQ4 Does the unified architecture reduce the pretrain-finetune discrepancy observed in existing models?
• RQ5 How does UniVL compare to existing state-of-the-art models on diverse video-text downstream tasks?

Key findings

• UniVL achieves state-of-the-art performance on five downstream video-text tasks, demonstrating strong generalization across understanding and generation benchmarks.
• The use of conditioned masked language and frame modeling objectives significantly improves cross-modal alignment and generation quality.
• The stage-by-stage pre-training strategy (StagedP) enhances model convergence and final performance compared to end-to-end training.
• The enhanced video representation strategy (EnhancedV) leads to more robust video feature learning, contributing to improved downstream results.
• The model reduces the pretrain-finetune discrepancy, showing better transferability to generation tasks than prior multimodal models.
• Empirical results on HowTo100M show that UniVL learns rich, transferable video-text representations that generalize across diverse tasks.