[Paper Review] VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

Hassan Akbari, Liangzhe Yuan+5 more|arXiv (Cornell University)|Apr 22, 2021

Human Pose and Action Recognition105 references340 citations

TL;DR

VATT trains convolution-free Transformers on raw video, audio, and text using multimodal contrastive losses, achieving state-of-the-art results in video action recognition and audio event classification without supervised pre-training.

ABSTRACT

We present a framework for learning multimodal representations from unlabeled data using convolution-free Transformer architectures. Specifically, our Video-Audio-Text Transformer (VATT) takes raw signals as inputs and extracts multimodal representations that are rich enough to benefit a variety of downstream tasks. We train VATT end-to-end from scratch using multimodal contrastive losses and evaluate its performance by the downstream tasks of video action recognition, audio event classification, image classification, and text-to-video retrieval. Furthermore, we study a modality-agnostic, single-backbone Transformer by sharing weights among the three modalities. We show that the convolution-free VATT outperforms state-of-the-art ConvNet-based architectures in the downstream tasks. Especially, VATT's vision Transformer achieves the top-1 accuracy of 82.1% on Kinetics-400, 83.6% on Kinetics-600, 72.7% on Kinetics-700, and 41.1% on Moments in Time, new records while avoiding supervised pre-training. Transferring to image classification leads to 78.7% top-1 accuracy on ImageNet compared to 64.7% by training the same Transformer from scratch, showing the generalizability of our model despite the domain gap between videos and images. VATT's audio Transformer also sets a new record on waveform-based audio event recognition by achieving the mAP of 39.4% on AudioSet without any supervised pre-training. VATT's source code is publicly available.

Motivation & Objective

• Motivate avoiding labeled data by leveraging large-scale multimodal video data.
• Develop a convolution-free Transformer architecture that processes raw video, audio, and text inputs.
• Propose a multimodal contrastive learning objective with a hierarchical common space for cross-modal alignment.
• Evaluate the learned representations on video action recognition, audio event classification, image classification, and text-to-video retrieval.

Proposed method

• Use modality-specific tokenization and separate positional encodings for video, audio, and text inputs.
• Adopt a convolution-free Transformer backbone with an aggregation token for sequence representation.
• Introduce DropToken to drop random token subsets during training to reduce computation.
• Construct a semantically hierarchical common space with projections g to align video, audio, and text via NCE and MIL-NCE losses.
• Train with multimodal contrastive objectives on HowTo100M (video-audio-text) and AudioSet (video-audio).
• Optionally share weights across modalities to form a modality-agnostic backbone (VATT-MA).

Experimental results

Research questions

• RQ1 Can a single convolution-free Transformer backbone learn from raw video, audio, and text via self-supervised multimodal objectives?
• RQ2 Does a modality-agnostic Transformer perform comparably to modality-specific backbones across tasks?
• RQ3 How does DropToken affect training efficiency and downstream performance in high-resolution multimodal data?
• RQ4 What are the transfer capabilities of VATT representations to image classification and zero-shot text-to-video retrieval?

Key findings

• VATT achieves top-1 accuracy of 82.1% on Kinetics-400 and 83.6% on Kinetics-600 when fine-tuned, with 41.1% on Moments in Time (all without supervised pre-training).
• VATT’s vision backbone pre-trained from multimodal data transfers to ImageNet with 78.7% top-1 accuracy, comparable to supervised-pretrained ViT variants.
• Audio Transformer fine-tuned on AudioSet achieves a mean average precision (mAP) of 39.4%, surpassing CNN-based baselines.
• Zero-shot text-to-video retrieval on YouCook2 and MSR-VTT using VATT’s video-text space achieves competitive results relative to prior multimodal methods, with batch size and epoch effects observed.
• A modality-agnostic backbone (VATT-MA) performs on par with modality-specific backbones for video action recognition after fine-tuning.
• DropToken significantly reduces pre-training computation while maintaining downstream performance, enabling high-resolution inputs.