[Paper Review] Masked Autoencoders As Spatiotemporal Learners

Christoph Feichtenhofer, Haoqi Fan+2 more|arXiv (Cornell University)|May 18, 2022

Generative Adversarial Networks and Image Synthesis243 citations

TL;DR

The paper extends Masked Autoencoders to spatiotemporal video data by masking spacetime patches and reconstructing pixels, achieving strong self-supervised representations with minimal domain bias and significant speedups.

ABSTRACT

This paper studies a conceptually simple extension of Masked Autoencoders (MAE) to spatiotemporal representation learning from videos. We randomly mask out spacetime patches in videos and learn an autoencoder to reconstruct them in pixels. Interestingly, we show that our MAE method can learn strong representations with almost no inductive bias on spacetime (only except for patch and positional embeddings), and spacetime-agnostic random masking performs the best. We observe that the optimal masking ratio is as high as 90% (vs. 75% on images), supporting the hypothesis that this ratio is related to information redundancy of the data. A high masking ratio leads to a large speedup, e.g., > 4x in wall-clock time or even more. We report competitive results on several challenging video datasets using vanilla Vision Transformers. We observe that MAE can outperform supervised pre-training by large margins. We further report encouraging results of training on real-world, uncurated Instagram data. Our study suggests that the general framework of masked autoencoding (BERT, MAE, etc.) can be a unified methodology for representation learning with minimal domain knowledge.

Motivation & Objective

• Investigate whether MAE can learn effective spatiotemporal video representations with minimal inductive biases.
• Examine the impact of high masking ratios on learning efficiency and performance on video tasks.
• Evaluate how MAE pre-training compares to supervised pre-training and explore scalability to real-world uncurated data.

Proposed method

• Extend MAE to videos by masking random spacetime patches and reconstructing pixels.
• Use vanilla Vision Transformers as encoder/decoder with patch and positional embeddings as the spacetime-aware components.
• Adopt spacetime-agnostic random masking to maximize information utilization and enable very high masking ratios.
• Train with mean squared error between predictions and original pixel targets for masked regions.
• Evaluate end-to-end by fine-tuning on downstream video recognition datasets (Kinetics-400, AVA, SSv2).
• Explore data preprocessing, mask sampling strategies, and decoder capacities to study efficiency and performance trade-offs.

Experimental results

Research questions

• RQ1 Does MAE pre-training on videos improve downstream video recognition performance compared to training from scratch?
• RQ2 What masking ratio and masking strategy yield the best trade-off between accuracy and computation for video MAE?
• RQ3 How does MAE pre-training on video data compare to supervised pre-training on the same or larger datasets?
• RQ4 Can MAE pre-training on real-world uncurated video data transfer effectively to standard video benchmarks?
• RQ5 What is the sensitivity of results to decoder size and reconstruction target choices?

Key findings

• MAE pre-training substantially improves video recognition accuracy (e.g., large gains over scratch) across multiple ViT sizes.
• Optimal masking ratio for video MAE is around 90% (up to 95% with longer training), enabling major encoder time/memory reductions (<1/10) and overall wall-clock speedups (≈4.1x) despite data-loading bottlenecks.
• Spacetime-agnostic masking outperforms structure-aware strategies, suggesting minimal spacetime inductive biases can learn effective representations from data.
• MAE pre-training on Kinetics (and other video datasets) can outperform supervised pre-training and even IN1K-based MAE pre-training on several downstream tasks (AVA, SSv2).
• Real-world Instagram video pre-training (un curated) yields competitive or superior transfer results versus curated datasets at scale, indicating practical potential for large-scale self-supervised video learning.