[Paper Review] VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling

Tsu-Jui Fu, Linjie Li+5 more|arXiv (Cornell University)|Nov 24, 2021

Multimodal Machine Learning ApplicationsComputer Science84 references89 citations

TL;DR

VIOLET introduces a fully end-to-end Video Swin Transformer-based VidL model with a novel Masked Visual-token Modeling pre-training task, achieving state-of-the-art results across multiple text-to-video retrieval and video QA benchmarks while modeling temporal dynamics explicitly.

ABSTRACT

A great challenge in video-language (VidL) modeling lies in the disconnection between fixed video representations extracted from image/video understanding models and downstream VidL data. Recent studies try to mitigate this disconnection via end-to-end training. To make it computationally feasible, prior works tend to "imagify" video inputs, i.e., a handful of sparsely sampled frames are fed into a 2D CNN, followed by a simple mean-pooling or concatenation to obtain the overall video representations. Although achieving promising results, such simple approaches may lose temporal information that is essential for performing downstream VidL tasks. In this work, we present VIOLET, a fully end-to-end VIdeO-LanguagE Transformer, which adopts a video transformer to explicitly model the temporal dynamics of video inputs. Further, unlike previous studies that found pre-training tasks on video inputs (e.g., masked frame modeling) not very effective, we design a new pre-training task, Masked Visual-token Modeling (MVM), for better video modeling. Specifically, the original video frame patches are "tokenized" into discrete visual tokens, and the goal is to recover the original visual tokens based on the masked patches. Comprehensive analysis demonstrates the effectiveness of both explicit temporal modeling via video transformer and MVM. As a result, VIOLET achieves new state-of-the-art performance on 5 video question answering tasks and 4 text-to-video retrieval tasks.

Motivation & Objective

• Motivate end-to-end VidL modeling to overcome disconnection from fixed video representations.
• Model temporal dynamics explicitly using a Video Swin Transformer.
• Introduce Masked Visual-token Modeling (MVM) to learn discrete visual tokens for video patches.
• Show that combining VT with cross-modal learning and MVM improves downstream VidL tasks.
• Demonstrate state-of-the-art results on multiple video QA and retrieval benchmarks.

Proposed method

• Use Video Swin Transformer to encode sparse-sampled video frames with explicit spatial-temporal modeling.
• Employ a Language Embedder to process text inputs and a Cross-modal Transformer to fuse video and text modalities.
• Pre-train with three tasks: Masked Language Modeling (MLM), Visual-Text Matching (VTM), and Masked Visual-token Modeling (MVM).
• MVM tokenizes frames via a discrete VAE (dVAE) and predicts masked visual tokens to reconstruct original patches.
• Adopt Blockwise Masking and Attended Masking to strengthen MLM and MVM signals by focusing on salient tokens/patches.
• Train end-to-end on image-text and video-text data, with curriculum including YT-Temporal, WebVid, and ConceptualCaptions datasets.

Experimental results

Research questions

• RQ1 Does explicit temporal modeling via a video transformer outperform simple mean pooling or concatenation of frame features for VidL tasks?
• RQ2 Does Masked Visual-token Modeling (MVM) provide measurable gains over prior visual masking strategies (MRM/MFM) in video-language pre-training?
• RQ3 How does joint pre-training on image-text and video-text data influence performance on text-to-video retrieval and video QA?
• RQ4 What is the impact of various pre-training data (WebVid, CC, YT-Temporal) on downstream VidL tasks?
• RQ5 Can end-to-end VidL training with MVM achieve state-of-the-art results across multiple benchmarks?

Key findings

• VIOLET achieves new state-of-the-art results on multiple text-to-video retrieval benchmarks and video QA datasets.
• Explicit temporal modeling with a Video Swin Transformer yields consistent gains over mean pooling and frame-concatenation baselines.
• Masked Visual-token Modeling (MVM) significantly improves downstream VidL performance across retrieval and QA tasks compared to MRM/MFM or MLM-based visual masking.
• Pre-training on WebVid+CC and YT-Temporal data provides strong improvements, with WebVid+CC delivering robust cross-modal learning signals.
• End-to-end training with MVM benefits even with relatively modest compute and frame resolutions, offering competitive performance with larger-scale approaches.