[Paper Review] Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling

Jie Lei, Linjie Li+5 more|arXiv (Cornell University)|Feb 11, 2021

Multimodal Machine Learning Applications81 references48 citations

TL;DR

ClipBERT enables end-to-end video-and-language learning by sparsely sampling short video clips during training, achieving comparable or better results than dense full-video feature methods across retrieval and question answering tasks.

ABSTRACT

The canonical approach to video-and-language learning (e.g., video question answering) dictates a neural model to learn from offline-extracted dense video features from vision models and text features from language models. These feature extractors are trained independently and usually on tasks different from the target domains, rendering these fixed features sub-optimal for downstream tasks. Moreover, due to the high computational overload of dense video features, it is often difficult (or infeasible) to plug feature extractors directly into existing approaches for easy finetuning. To provide a remedy to this dilemma, we propose a generic framework ClipBERT that enables affordable end-to-end learning for video-and-language tasks, by employing sparse sampling, where only a single or a few sparsely sampled short clips from a video are used at each training step. Experiments on text-to-video retrieval and video question answering on six datasets demonstrate that ClipBERT outperforms (or is on par with) existing methods that exploit full-length videos, suggesting that end-to-end learning with just a few sparsely sampled clips is often more accurate than using densely extracted offline features from full-length videos, proving the proverbial less-is-more principle. Videos in the datasets are from considerably different domains and lengths, ranging from 3-second generic domain GIF videos to 180-second YouTube human activity videos, showing the generalization ability of our approach. Comprehensive ablation studies and thorough analyses are provided to dissect what factors lead to this success. Our code is publicly available at https://github.com/jayleicn/ClipBERT

Motivation & Objective

• Motivate end-to-end video-and-language learning without relying on fixed, densely extracted features.
• Propose a sparse sampling framework that trains on few short clips per video to reduce memory and computation.
• Demonstrate that image-text pre-training benefits video-text tasks and that end-to-end finetuning improves performance.

Proposed method

• We sparsely sample one or a few short clips per video during training and aggregate predictions from sampled clips.
• Each sampled clip is encoded with a 2D ResNet-50 vision backbone and fused with text via a 12-layer transformer in a cross-modal module.
• During training, gradients flow through vision and language encoders for task-specific finetuning (end-to-end learning).
• During inference, predictions from densely sampled clips are aggregated (e.g., mean, LogSumExp) to form the final video-level output.
• We initialize vision and language components from image-text pre-training (COCO/Visual Genome) and finetune on downstream video-text tasks.

Experimental results

Research questions

• RQ1 Does sparse sampling at training provide competitive or superior performance to dense full-video feature methods across text-to-video retrieval and video question answering?
• RQ2 Can image-text pre-training and end-to-end finetuning improve video-text understanding tasks?
• RQ3 What are the effects of input resolution, number of frames, and number of clips on performance and efficiency?
• RQ4 How does the aggregation strategy over clips affect final predictions?
• RQ5 Is sparse sampling more memory-efficient than dense sampling while maintaining or improving accuracy?

Key findings

• Sparse sampling with two frames per clip often matches or surpasses dense sampling of 16 frames in both retrieval and QA tasks.
• Mean pooling over frames is more effective than 3D convolutions for fusing temporal information in ClipBERT.
• Increasing inference clips improves performance up to a point, with diminishing returns beyond several clips.
• Sparse random sampling outperforms dense uniform sampling in memory and computation efficiency while delivering better or comparable accuracy.
• Image-text pre-training (COCO/Visual Genome) significantly boosts MSRVTT retrieval and MSRVTT-QA results after end-to-end finetuning.
• End-to-end training yields better performance than freezing either vision or language encoders.