[Paper Review] Self-supervised Video Representation Learning by Pace Prediction

Jiangliu Wang, Jianbo Jiao+1 more|arXiv (Cornell University)|Aug 13, 2020

Human Pose and Action Recognition2 references61 citations

TL;DR

Proposes pace prediction as a self-supervised pretext task for learning video representations without motion channels, enhanced with contrastive learning; achieves state-of-the-art results on action recognition and video retrieval across multiple backbones.

ABSTRACT

This paper addresses the problem of self-supervised video representation learning from a new perspective -- by video pace prediction. It stems from the observation that human visual system is sensitive to video pace, e.g., slow motion, a widely used technique in film making. Specifically, given a video played in natural pace, we randomly sample training clips in different paces and ask a neural network to identify the pace for each video clip. The assumption here is that the network can only succeed in such a pace reasoning task when it understands the underlying video content and learns representative spatio-temporal features. In addition, we further introduce contrastive learning to push the model towards discriminating different paces by maximizing the agreement on similar video content. To validate the effectiveness of the proposed method, we conduct extensive experiments on action recognition and video retrieval tasks with several alternative network architectures. Experimental evaluations show that our approach achieves state-of-the-art performance for self-supervised video representation learning across different network architectures and different benchmarks. The code and pre-trained models are available at https://github.com/laura-wang/video-pace.

Motivation & Objective

• Motivate self-supervised video representation learning using video pace sensitivity analogous to human perception.
• Introduce a pace prediction pretext task that uses randomly sampled clips at different paces to learn spatio-temporal features.
• Enhance the pace task with contrastive learning to regularize and improve discriminative power.
• Evaluate across multiple backbones (C3D, 3D-ResNet, R(2+1)D, S3D-G) on action recognition and video retrieval.
• Demonstrate the method’s effectiveness and potential to scale with unlabeled video data.

Proposed method

• Sample video clips at multiple pacing rates from unlabeled videos to create a pace prediction pretext task.
• Train a 3D CNN backbone to classify the pace applied to each input clip using a cross-entropy loss.
• Incorporate contrastive learning to maximize agreement between positive pairs (same pace or same context) and separate negatives.
• Investigate two contrastive configurations: same context (content-aware) and same pace (content-agnostic), and their impact on performance.
• Combine pace prediction loss with contrastive loss via a weighted sum objective.
• Evaluate with several backbones (C3D, 3D-ResNet, R(2+1)D, S3D-G) and on downstream tasks like action recognition and video retrieval.

Experimental results

Research questions

• RQ1 Can a pace-based pretext task enable learning powerful spatio-temporal video representations without motion channels?
• RQ2 Does adding contrastive learning further improve representations learned via pace prediction?
• RQ3 How do different backbone architectures respond to pace-based self-supervision?
• RQ4 What is the impact of same-context versus same-pace contrastive strategies on downstream performance?
• RQ5 How do the proposed methods perform on standard video understanding benchmarks (action recognition and retrieval) when pre-trained on unlabeled data?

Key findings

• Pace prediction alone yields strong improvements over random initialization across multiple backbones.
• Incorporating contrastive learning further boosts performance, with same-context contrast generally outperforming same-pace in many setups.
• R(2+1)D backbone with pace prediction achieves top results among evaluated configurations on UCF101 and HMDB51.
• The combination of pace prediction and context-based contrastive learning yields state-of-the-art or competitive results against contemporary self-supervised methods.
• Attention visualizations indicate the model focuses on motion regions when trained with pace-based supervision, supporting the learned spatio-temporal reasoning.
• The approach demonstrates strong performance across action recognition and video retrieval tasks using only video modality.