[Paper Review] MERLOT: Multimodal Neural Script Knowledge Models

Rowan Zellers, Ximing Lu+6 more|arXiv (Cornell University)|Jun 4, 2021

Multimodal Machine Learning Applications125 references54 citations

TL;DR

MERLOT learns multimodal script knowledge from 6M YouTube videos using self-supervised objectives, achieving state-of-the-art results on 12 video QA tasks and transferring to images.

ABSTRACT

As humans, we understand events in the visual world contextually, performing multimodal reasoning across time to make inferences about the past, present, and future. We introduce MERLOT, a model that learns multimodal script knowledge by watching millions of YouTube videos with transcribed speech -- in an entirely label-free, self-supervised manner. By pretraining with a mix of both frame-level (spatial) and video-level (temporal) objectives, our model not only learns to match images to temporally corresponding words, but also to contextualize what is happening globally over time. As a result, MERLOT exhibits strong out-of-the-box representations of temporal commonsense, and achieves state-of-the-art performance on 12 different video QA datasets when finetuned. It also transfers well to the world of static images, allowing models to reason about the dynamic context behind visual scenes. On Visual Commonsense Reasoning, MERLOT answers questions correctly with 80.6% accuracy, outperforming state-of-the-art models of similar size by over 3%, even those that make heavy use of auxiliary supervised data (like object bounding boxes). Ablation analyses demonstrate the complementary importance of: 1) training on videos versus static images; 2) scaling the magnitude and diversity of the pretraining video corpus; and 3) using diverse objectives that encourage full-stack multimodal reasoning, from the recognition to cognition level.

Motivation & Objective

• Motivate learning commonsense, temporal reasoning, and multimodal world knowledge from unlabeled video-and-speech data.
• Develop a self-supervised pretraining framework that aligns video frames with transcripts and contextualizes them over time.
• Evaluate transferability to downstream vision-language tasks including video QA and static image reasoning.
• Release a diverse YouTube-derived corpus and open-source models to enable research on multimodal temporal reasoning.

Proposed method

• Pretrain MERLOT on 6 million unlabeled YouTube videos using a sequence of video segments; each segment includes a frame and spoken transcript segment.
• Use a joint vision-language Transformer to encode frames and transcript segments and learn cross-modal representations.
• Three pretraining objectives: (i) contrastive frame-transcript matching to align frames with their contextualized transcripts, (ii) attention-masked masked language modeling to reconstruct grounded words, (iii) temporal reordering to learn event ordering over time.
• Train on YT-Temporal-180M, a diverse corpus; use a grid-based image encoder (ResNet-50 + Vision Transformer) and a 12-layer RoBERTa-style joint encoder.
• Fine-tune on downstream tasks across 14 datasets, including 12 video reasoning benchmarks and VCR (visual commonsense reasoning).
• Release code, data, and models for research use.

Experimental results

Research questions

• RQ1 Can multimodal script knowledge be learned from unlabeled videos and transcripts without manual annotations?
• RQ2 Do frame-level and video-level objectives complement each other to improve temporal and multimodal reasoning?
• RQ3 How well do video-trained representations transfer to static image reasoning tasks requiring temporal or narrative understanding?
• RQ4 What is the impact of data diversity, pretraining duration, and objective design on downstream performance?

Key findings

• MERLOT achieves state-of-the-art results on 12 downstream video reasoning tasks when finetuned.
• On Visual Commonsense Reasoning (VCR), MERLOT attains 80.6% accuracy, outperforming comparable baselines by more than 3%.
• In Table 1, MERLOT (base-sized) reaches Spearman 0.733, Pairwise accuracy 84.5, and Distance 0.498, surpassing CLIP (0.609, 78.7, 0.638) and UNITER (0.545, 75.2, 0.745).
• Pretraining on videos (not just images) and increasing video diversity improves performance; longer pretraining yields continuous gains.
• MERLOT transfers to static images and demonstrates temporal commonsense reasoning in tasks like unscrambling visual stories, outperforming baselines that rely on image-caption pairs.