[Paper Review] Spatiotemporal Contrastive Video Representation Learning

Rui Qian, Tianjian Meng+5 more|arXiv (Cornell University)|Aug 9, 2020

Human Pose and Action RecognitionComputer Science76 references84 citations

TL;DR

A self-supervised contrastive learning framework (CVRL) for videos that jointly leverages temporally consistent spatial augmentations and a sampling-based temporal augmentation to learn powerful spatiotemporal representations from unlabeled videos, evaluated with linear probes and downstream tasks.

ABSTRACT

We present a self-supervised Contrastive Video Representation Learning (CVRL) method to learn spatiotemporal visual representations from unlabeled videos. Our representations are learned using a contrastive loss, where two augmented clips from the same short video are pulled together in the embedding space, while clips from different videos are pushed away. We study what makes for good data augmentations for video self-supervised learning and find that both spatial and temporal information are crucial. We carefully design data augmentations involving spatial and temporal cues. Concretely, we propose a temporally consistent spatial augmentation method to impose strong spatial augmentations on each frame of the video while maintaining the temporal consistency across frames. We also propose a sampling-based temporal augmentation method to avoid overly enforcing invariance on clips that are distant in time. On Kinetics-600, a linear classifier trained on the representations learned by CVRL achieves 70.4% top-1 accuracy with a 3D-ResNet-50 (R3D-50) backbone, outperforming ImageNet supervised pre-training by 15.7% and SimCLR unsupervised pre-training by 18.8% using the same inflated R3D-50. The performance of CVRL can be further improved to 72.9% with a larger R3D-152 (2x filters) backbone, significantly closing the gap between unsupervised and supervised video representation learning. Our code and models will be available at https://github.com/tensorflow/models/tree/master/official/.

Motivation & Objective

• Motivate learning powerful video representations from unlabeled data.
• Investigate how spatial and temporal augmentations affect self-supervised video learning.
• Develop augmentation strategies that preserve temporal coherence while enforcing useful invariances.
• Demonstrate scalability with larger backbones and more data to close the gap with supervised video pre-training.

Proposed method

• Use a contrastive loss (InfoNCE) to pull together two augmented clips from the same video and push apart clips from different videos.
• Encode clips with a 3D-ResNet backbone (R3D-50 as default) with a projection head producing 128-dimensional features for contrastive learning.
• Design temporally consistent spatial augmentations that apply fixed randomness across frames to preserve motion cues.
• Implement a sampling-based temporal augmentation where the time interval between two clips follows a monotonically decreasing distribution to emphasize near-temporal clips.
• Train with large mini-batches and synchronized batch normalization; evaluate with linear probes on frozen backbones and with fine-tuning for semi-supervised and downstream tasks.

Experimental results

Research questions

• RQ1 Can spatiotemporal augmentations improve self-supervised video representation learning compared to spatial-only or temporal-only cues?
• RQ2 What augmentation strategies best preserve temporal dynamics while enabling strong spatial invariance?
• RQ3 How does CVRL scale with larger backbones and more unlabeled data on standard video benchmarks?
• RQ4 How do CVRL representations transfer to semi-supervised learning, action classification, and detection tasks?

Key findings

• CVRL yields substantial gains in linear evaluation, e.g., 70.4% top-1 on Kinetics-600 with R3D-50 and 72.9% with R3D-152, closing the gap to supervised methods.
• CVRL outperforms ImageNet-inflated and SimCLR-inflated baselines under the same inflated 3D backbones, with 15%+ gains over ImageNet pre-training and 18.8% over SimCLR in linear evaluation on K400.
• Combining temporally consistent spatial augmentations with a monotonic decreasing temporal sampling distribution yields large performance gains (e.g., 63.8% top-1 with CVRL vs 52.3% using both augmentations without temporal consistency).
• More data and larger backbones consistently improve CVRL performance (e.g., 200 to 800 pre-training epochs and R3D-152 showing strong results on K400/K600).
• In downstream tasks, CVRL achieves competitive/strong results on UCF-101, HMDB-51, and AVA action detection, often surpassing single-modality and parity with multimodal baselines when pre-trained on K400–K600.