[Paper Review] VideoBERT: A Joint Model for Video and Language Representation Learning

Chen Sun, Austin Myers+3 more|arXiv (Cornell University)|Apr 3, 2019

Multimodal Machine Learning ApplicationsComputer Science38 references149 citations

TL;DR

VideoBERT trains a joint visual-linguistic model by quantizing video features into visual words and pairing them with ASR-derived text using a BERT-like transformer, enabling open-vocabulary video understanding, zero-shot action classification, and state-of-the-art video captioning on YouCook II.

ABSTRACT

Self-supervised learning has become increasingly important to leverage the abundance of unlabeled data available on platforms like YouTube. Whereas most existing approaches learn low-level representations, we propose a joint visual-linguistic model to learn high-level features without any explicit supervision. In particular, inspired by its recent success in language modeling, we build upon the BERT model to learn bidirectional joint distributions over sequences of visual and linguistic tokens, derived from vector quantization of video data and off-the-shelf speech recognition outputs, respectively. We use VideoBERT in numerous tasks, including action classification and video captioning. We show that it can be applied directly to open-vocabulary classification, and confirm that large amounts of training data and cross-modal information are critical to performance. Furthermore, we outperform the state-of-the-art on video captioning, and quantitative results verify that the model learns high-level semantic features.

Motivation & Objective

• Demonstrate that high-level semantic video representations can be learned without manual labels by leveraging joint video-language modeling.
• Adapt BERT to handle discrete visual tokens alongside linguistic tokens to learn p(x, y) over video and language sequences.
• Show that VideoBERT supports open-vocabulary action classification and outperforms prior methods on video captioning tasks.
• Investigate the impact of large-scale pretraining data and cross-modal information on performance.
• Explore the transferability of VideoBERT features for downstream tasks like captioning.

Proposed method

• Represent video as sequences of discrete visual words obtained via hierarchical vector quantization of 3D video features.
• Use a BERT-style masked token objective to learn bidirectional dependencies in sequences containing both visual and linguistic tokens.
• Incorporate a linguistic-visual alignment objective to teach text-video correspondence, using [CLS] representations to predict alignment.
• Train on a large YouTube cooking video dataset with English ASR for text supervision, initialized from a BERT-LARGE checkpoint.
• Evaluate as a probabilistic model for zero-shot action classification and as a feature extractor for captioning tasks.
• Demonstrate that increasing pretraining data size yields monotonic performance gains and that cross-modal pretraining improves captioning.

Experimental results

Research questions

• RQ1 Can a joint visual-linguistic model learn high-level semantic video representations without manual labels?
• RQ2 How well can a BERT-based model handle discrete visual tokens together with linguistic tokens to perform open-vocabulary tasks?
• RQ3 Does cross-modal training (video and text) improve video captioning compared to video-only training?
• RQ4 How does the size of pretraining data affect the learned representations and downstream performance?
• RQ5 Can VideoBERT features transfer effectively to video captioning and other downstream tasks?

Key findings

• VideoBERT achieves competitive open-vocabulary action classification on YouCook II, outperforming text-only and language-prior baselines.
• Increasing pretraining data size (10K to 300K videos) yields monotonic gains in verb/object recognition metrics.
• VideoBERT outperforms the S3D baseline on YouCook II captioning metrics, and cross-modal VideoBERT (video+text) surpasses video-only variants.
• Combining VideoBERT with S3D features yields the best captioning performance across BLEU, METEOR, ROUGE-L, and CIDEr.
• Zero-shot captioning capabilities show VideoBERT can leverage learned visual-language representations without YouCook II supervision.