[Paper Review] Self-Supervised MultiModal Versatile Networks

Jean-Baptiste Alayrac, Adrià Recasens+7 more|arXiv (Cornell University)|Jun 29, 2020

Multimodal Machine Learning Applications92 references195 citations

TL;DR

The paper introduces Self-Supervised MultiModal Versatile (MMV) networks that learn joint visual, audio, and language representations from unlabelled videos, with a deflation mechanism to apply the model to still images and strong zero-shot and supervised transfer performance.

ABSTRACT

Videos are a rich source of multi-modal supervision. In this work, we learn representations using self-supervision by leveraging three modalities naturally present in videos: visual, audio and language streams. To this end, we introduce the notion of a multimodal versatile network -- a network that can ingest multiple modalities and whose representations enable downstream tasks in multiple modalities. In particular, we explore how best to combine the modalities, such that fine-grained representations of the visual and audio modalities can be maintained, whilst also integrating text into a common embedding. Driven by versatility, we also introduce a novel process of deflation, so that the networks can be effortlessly applied to the visual data in the form of video or a static image. We demonstrate how such networks trained on large collections of unlabelled video data can be applied on video, video-text, image and audio tasks. Equipped with these representations, we obtain state-of-the-art performance on multiple challenging benchmarks including UCF101, HMDB51, Kinetics600, AudioSet and ESC-50 when compared to previous self-supervised work. Our models are publicly available.

Motivation & Objective

• Motivate learning generic, versatile multimodal representations from unlabelled video data.
• Develop a network that can ingest vision, audio, or text and compare across modalities.
• Respect modality-specific granularity, enabling fine-grained visual/audio similarity and coarse-grained text alignment.
• Enable efficient application to both video streams and static images through a deflation mechanism.

Proposed method

• Embed each modality into a shared or hierarchical space using modality-specific backbones and projection heads.
• Investigate three modality embedding graphs (Shared, Disjoint, Fine-and-Coarse FAC) to align modalities in joint spaces.
• Train with a multimodal contrastive loss that enforces positive pairs from the same video and negative pairs from different videos.
• Use MIL-NCE for text alignment to handle narration misalignment with video content.
• Introduce a deflation procedure to convert a video-trained network into an image-powered network without labels.
• Handle missing modalities by omitting the corresponding loss term and reweighting remaining losses.

Experimental results

Research questions

• RQ1 Can a single multimodal network effectively integrate visual, auditory, and textual information learned from unlabelled videos?
• RQ2 Which modality embedding graph yields the best trade-offs between cross-modal alignment, intra-modality granularity, and cross-modal navigability?
• RQ3 Does deflating a video-trained network provide competitive image representations without additional supervision?
• RQ4 How do three-modality models compare to two-modality baselines on standard video, audio, and image benchmarks?

Key findings

• The FAC (Fine and Coarse) embedding strategy yields strong performance across UCF101, HMDB51, MSRVTT, and ESC-50, outperforming two-modality configurations.
• Training with all three modalities generally improves visual representations and supports cross-modal retrieval tasks.
• Combining HowTo100M with AudioSet improves HMDB51, UCF101, and ESC-50, and enables better use of audio data in the absence of text.
• The deflated video-to-image network enables evaluation on image tasks with competitive performance, without needing new annotations.
• The approach achieves state-of-the-art results among self-supervised methods on several benchmarks and approaches supervised performance on large-scale tasks like Kinetics600.