[Paper Review] SMAUG: Sparse Masked Autoencoder for Efficient Video-Language Pre-training

Yuanze Lin, Chen Wei+3 more|arXiv (Cornell University)|Nov 21, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

SMAUG proposes a sparse masked autoencoder framework for efficient video-language pre-training by jointly masking visual and textual tokens and applying space-time token sparsification to reduce computational cost. It achieves state-of-the-art or competitive performance on text-to-video retrieval and video question answering across six benchmarks, with a 1.9× speedup—pre-training in only ~50 NVIDIA A6000 GPU hours.

ABSTRACT

Video-language pre-training is crucial for learning powerful multi-modal representation. However, it typically requires a massive amount of computation. In this paper, we develop SMAUG, an efficient pre-training framework for video-language models. The foundation component in SMAUG is masked autoencoders. Different from prior works which only mask textual inputs, our masking strategy considers both visual and textual modalities, providing a better cross-modal alignment and saving more pre-training costs. On top of that, we introduce a space-time token sparsification module, which leverages context information to further select only "important" spatial regions and temporal frames for pre-training. Coupling all these designs allows our method to enjoy both competitive performances on text-to-video retrieval and video question answering tasks, and much less pre-training costs by 1.9X or more. For example, our SMAUG only needs about 50 NVIDIA A6000 GPU hours for pre-training to attain competitive performances on these two video-language tasks across six popular benchmarks.

Motivation & Objective

• Reduce the massive computational cost of video-language pre-training, which currently requires weeks of training on multiple GPUs.
• Address redundancy in video data by identifying and removing uninformative spatial patches and temporally irrelevant frames during pre-training.
• Improve efficiency without sacrificing performance by jointly masking visual and textual modalities in a masked autoencoder framework.
• Enable scalable and cost-effective pre-training that maintains strong transfer performance on downstream tasks like text-to-video retrieval and video question answering.

Proposed method

• Apply masked autoencoders (MAE) to both visual and textual modalities, masking up to 75% of patches and frames to reduce computation during pre-training.
• Introduce a space-time token sparsification module that uses attention weights to identify and retain only the most informative spatial patches per frame.
• Implement a learnable Transformer-based selector to dynamically select key frames from a video clip, reducing temporal redundancy.
• Train the model using masked visual modeling (MVM) and masked language modeling (MLM), reconstructing both masked visual patches and text tokens.
• Use a lightweight decoder to reconstruct masked patches and tokens, enabling efficient training on only visible, salient tokens.
• Combine joint masking and sparsification to reduce pre-training time while preserving cross-modal alignment and representation quality.

Experimental results

Research questions

• RQ1 Can joint masking of visual and textual modalities in a masked autoencoder framework significantly reduce pre-training cost while maintaining performance?
• RQ2 To what extent can spatial and temporal redundancy in video data be removed without degrading model performance?
• RQ3 How effective is a learnable frame selector in identifying key frames for pre-training, and what is its impact on computational efficiency?
• RQ4 Can the proposed sparsification strategy achieve a 1.9× speedup in pre-training time while maintaining state-of-the-art performance across multiple benchmarks?

Key findings

• SMAUG reduces pre-training time by 1.9×, achieving competitive performance on six video-language benchmarks with only ~50 NVIDIA A6000 GPU hours.
• Using a 0.8 keeping rate in visual token sparsification reduces pre-training time from 53.8 to 50.4 hours with only a 0.5% drop in R@1 on MSRVTT.
• Selecting 2 out of 4 frames (4→2) reduces pre-training time from 82.5 to 77.8 hours, with only a 0.5% performance drop on R@1.
• The full SMAUG pipeline (MAE + VTS + FS) reduces pre-training time from 144.5 to 77.8 hours (1.9× speedup), losing only 1.4% on R@1.
• On a 17M video-text corpus, SMAUG achieves a 2.5% gain in R@1 over Singularity while saving 87.1 GPU hours, demonstrating strong scalability.
• Qualitative results show the frame selector effectively identifies relevant frames aligned with captions, and the model produces reliable pixel reconstructions even at 50% masking ratio.