[Paper Review] Parameter Efficient Multimodal Transformers for Video Representation Learning

Sang-Ho Lee, Youngjae Yu+4 more|arXiv (Cornell University)|Dec 8, 2020

Music and Audio Processing82 references36 citations

TL;DR

This paper presents end-to-end trainable multimodal transformers for audio-visual video representation with aggressive parameter sharing and low-rank decomposition, achieving up to 97% parameter reduction and introducing content-aware negative sampling and fusion strategy analyses.

ABSTRACT

The recent success of Transformers in the language domain has motivated adapting it to a multimodal setting, where a new visual model is trained in tandem with an already pretrained language model. However, due to the excessive memory requirements from Transformers, existing work typically fixes the language model and train only the vision module, which limits its ability to learn cross-modal information in an end-to-end manner. In this work, we focus on reducing the parameters of multimodal Transformers in the context of audio-visual video representation learning. We alleviate the high memory requirement by sharing the parameters of Transformers across layers and modalities; we decompose the Transformer into modality-specific and modality-shared parts so that the model learns the dynamics of each modality both individually and together, and propose a novel parameter sharing scheme based on low-rank approximation. We show that our approach reduces parameters of the Transformers up to 97$\%$, allowing us to train our model end-to-end from scratch. We also propose a negative sampling approach based on an instance similarity measured on the CNN embedding space that our model learns together with the Transformers. To demonstrate our approach, we pretrain our model on 30-second clips (480 frames) from Kinetics-700 and transfer it to audio-visual classification tasks.

Motivation & Objective

• Motivate learning long-term audio-visual representations from unlabeled videos using transformers.
• Enable end-to-end training by reducing memory and parameter requirements.
• Investigate parameter-sharing schemes across transformers and layers.
• Propose effective negative sampling to improve self-supervised cross-modal learning.
• Evaluate fusion strategies and demonstrate downstream task transfer to short and long videos.

Proposed method

• A three-part model: visual/audio CNNs for short-term features, unimodal Transformers for long-term context, and a multimodal Transformer for cross-modal context.
• Parameter reduction via shared, low-rank transformer weights across modalities and across layers, decomposed as W=UΣVᵀ with shared U and private ΣVᵀ.
• Positional BOS tokens and time embeddings to preserve temporal order in unimodal streams.
• Modality-shared and time-shared embeddings in the multimodal Transformer to enable cross-modal fusion.
• Self-supervised pretraining with two tasks: Masked Embedding Prediction (MEP) using InfoNCE and Correct Pair Prediction (CPP) for cross-modal correspondence.
• Content-aware negative sampling (CANS) that selects negatives based on CNN embedding similarities within a mini-batch.

Experimental results

Research questions

• RQ1 Can a parameter-efficient Transformer architecture learn end-to-end audio-visual video representations from scratch?
• RQ2 How does cross-modal fusion strategy affect multimodal representation learning and downstream performance?
• RQ3 What is the impact of sharing transformer weights across modalities and layers on model size and accuracy?
• RQ4 Does content-aware negative sampling improve self-supervised learning for multimodal video representations?
• RQ5 How do pretrained multimodal representations transfer to short- and long-duration video classification tasks?

Key findings

• Up to 97% reduction in Transformer parameters without drastic performance loss (128M down to 4M with Part sharing).
• Mid-level fusion consistently yields strong audio-visual performance and robustness to missing modalities. In ablations, Mid achieved 65.7% top-1 and 89.9% top-5 on audio-visual classification, outperforming Early and Late fusion in certain settings.
• Content-aware negative sampling (CANS-Similar) improves MEP, achieving 67.5% top-1 and 92.3% top-5 in multimodal results in Table 1.
• Cross-layer weight sharing is effective; sharing across layers does not hurt performance, enabling smaller, faster models.
• Pretraining on Kinetics-700 or AudioSet with Mid fusion and CANS-Similar yields strong short- and long-duration audio-visual results, outperforming several baselines on multiple datasets.