[Paper Review] Support-set bottlenecks for video-text representation learning

Mandela Patrick, Po-Yao Huang+5 more|arXiv (Cornell University)|Oct 6, 2020

Multimodal Machine Learning ApplicationsComputer Science96 references37 citations

TL;DR

The paper introduces cross-instance captioning with a support-set bottleneck to complement contrastive video-text learning, improving semantic sharing and retrieval performance across multiple datasets.

ABSTRACT

The dominant paradigm for learning video-text representations -- noise contrastive learning -- increases the similarity of the representations of pairs of samples that are known to be related, such as text and video from the same sample, and pushes away the representations of all other pairs. We posit that this last behaviour is too strict, enforcing dissimilar representations even for samples that are semantically-related -- for example, visually similar videos or ones that share the same depicted action. In this paper, we propose a novel method that alleviates this by leveraging a generative model to naturally push these related samples together: each sample's caption must be reconstructed as a weighted combination of other support samples' visual representations. This simple idea ensures that representations are not overly-specialized to individual samples, are reusable across the dataset, and results in representations that explicitly encode semantics shared between samples, unlike noise contrastive learning. Our proposed method outperforms others by a large margin on MSR-VTT, VATEX and ActivityNet, and MSVD for video-to-text and text-to-video retrieval.

Motivation & Objective

• Motivate improving video-text representations beyond strict instance discrimination

Proposed method

• Combine cross-modal contrastive learning with a generative cross-captioning objective
• Introduce a cross-instance attention mechanism that reconstructs captions from a weighted mix of other videos in the batch
• Define a.batch-level attention to select a support set and form reconstructed text representations
• Use a hinge-based triplet contrastive loss for video-text pairs and a cross-captioning loss with a tunable weight lambda
• Experiment with variants of the cross-captioning attention (Identity, Full, Hybrid, Cross) and study the impact of support-set size
• Train with Adam and freeze the video encoder while finetuning others

Experimental results

Research questions

• RQ1 Can a generative cross-captioning objective improve multimodal representations learned with a contrastive loss?
• RQ2 Does reconstructing captions from a batch-based support set encourage semantic sharing across samples?
• RQ3 Which cross-captioning variant provides the best retrieval performance across datasets?
• RQ4 How does the size of the support set affect retrieval performance?
• RQ5 What is the impact of pretraining on HowTo100M on final results?

Key findings

• The Cross variant of cross-captioning yields the best text-to-video retrieval on MSR-VTT (27.2% R@1, 55.2% R@5) and related metrics
• In ablations, combining temporal attention, stronger text encoding/decoding, and a triplet-based contrastive loss improves results over baselines
• Pretraining on HowTo100M further boosts performance across MSR-VTT, VATEX, ActivityNet, and MSVD
• The cross-captioning objective acts as a bottleneck encouraging concept sharing and improves semantic retrieval
• Smaller and excessively large support sets degrade performance, indicating an optimal intermediate size
• Qualitative attention analyses show the model focuses on semantically related samples rather than memorizing isolated video-caption pairs