[Paper Review] A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning

Christoph Feichtenhofer, Haoqi Fan+3 more|arXiv (Cornell University)|Apr 29, 2021

Human Pose and Action Recognition94 references19 citations

TL;DR

This paper proposes a simple yet effective unsupervised spatiotemporal representation learning method that encourages temporal feature persistency across clips in videos, generalizing image-based contrastive learning frameworks (MoCo, SimCLR, BYOL, SwAV) to video with minimal modification. It achieves state-of-the-art performance on multiple benchmarks, including surpassing supervised pre-training in select cases, and demonstrates that long-span (up to 60-second) temporal consistency significantly improves downstream accuracy.

ABSTRACT

We present a large-scale study on unsupervised spatiotemporal representation learning from videos. With a unified perspective on four recent image-based frameworks, we study a simple objective that can easily generalize all these methods to space-time. Our objective encourages temporally-persistent features in the same video, and in spite of its simplicity, it works surprisingly well across: (i) different unsupervised frameworks, (ii) pre-training datasets, (iii) downstream datasets, and (iv) backbone architectures. We draw a series of intriguing observations from this study, e.g., we discover that encouraging long-spanned persistency can be effective even if the timespan is 60 seconds. In addition to state-of-the-art results in multiple benchmarks, we report a few promising cases in which unsupervised pre-training can outperform its supervised counterpart. Code is made available at https://github.com/facebookresearch/SlowFast

Motivation & Objective

• To investigate whether a simple, unified objective for temporal feature persistency can generalize across diverse unsupervised video representation learning frameworks.
• To evaluate the effectiveness of unsupervised pre-training on large-scale video datasets, including Kinetics and Instagram, across multiple downstream tasks.
• To identify key hyperparameters such as temporal span, number of clips, and data curation that influence representation quality.
• To compare unsupervised pre-training performance against supervised pre-training in video action recognition and detection tasks.
• To provide a comprehensive ablation study on backbone architectures, data augmentation, and training duration for unsupervised video learning.

Proposed method

• The method generalizes image contrastive learning by replacing image crops with temporal clips from the same video, encouraging similarity between representations of different clips from the same video.
• A unified contrastive objective is applied across four frameworks—MoCo, SimCLR, BYOL, and SwAV—using either negative samples or momentum encoders, with minimal architectural changes.
• The approach uses a positive pair loss that maximizes similarity between features of temporally shifted clips (e.g., query and key clips) from the same video, promoting temporal persistence.
• The method is trained on large-scale datasets: Kinetics-400 (240k videos) and three million-scale Instagram video sets, both curated and uncurated.
• Backbone models include ResNeXt-101 and ResNet-50, with feature extraction and fine-tuning performed using standard linear probing and end-to-end adaptation on downstream tasks.
• The framework supports both supervised and unsupervised pre-training, with ablation studies on data type (trimmed vs. untrimmed), augmentation, and training duration.

Experimental results

Research questions

• RQ1 Can a simple temporal persistency objective generalize across multiple unsupervised video learning frameworks (MoCo, SimCLR, BYOL, SwAV) without architectural changes?
• RQ2 How does the length of the temporal span between positive clips (e.g., 1s, 30s, 60s) affect downstream performance in video representation learning?
• RQ3 Does unsupervised pre-training on large-scale video data outperform or match supervised pre-training on downstream action recognition and detection tasks?
• RQ4 What is the impact of data curation (curated vs. uncurated), data type (Kinetics vs. Instagram), and training duration on representation quality?
• RQ5 How do factors like number of clips per video, contrastive loss design, and momentum encoder usage influence the final performance?

Key findings

• The proposed temporal persistency objective achieves state-of-the-art performance on UCF-101, HMDB51, and Kinetics-400 under unsupervised pre-training, with top-1 accuracy of 88.4% on Kinetics-400.
• Encouraging long-span temporal consistency (up to 60 seconds) significantly improves downstream performance, demonstrating that long-range temporal modeling is effective even without explicit supervision.
• In several cases, unsupervised pre-training outperforms supervised pre-training, particularly on Something-Something V2 and AVA, indicating that self-supervised learning can capture richer spatiotemporal invariances.
• Using multiple clips per video (ρ > 1) and longer temporal spans consistently improves representation quality across all frameworks and datasets.
• The method generalizes effectively across different backbones (ResNeXt-101, ResNet-50), data types (curated Kinetics, uncurated Instagram), and augmentation strategies.
• Ablation studies show that momentum encoders and contrastive objectives improve performance, while uncurated data still yields strong results, suggesting robustness to data distribution shifts.