[Paper Review] Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction

Longlong Jing, Xiaodong Yang+2 more|arXiv (Cornell University)|Nov 28, 2018

Human Pose and Action Recognition61 references143 citations

TL;DR

This paper introduces 3DRotNet, a fully self-supervised 3D CNN that learns spatiotemporal video features by predicting rotation transformations, enabling strong transfer to action recognition on small datasets.

ABSTRACT

The success of deep neural networks generally requires a vast amount of training data to be labeled, which is expensive and unfeasible in scale, especially for video collections. To alleviate this problem, in this paper, we propose 3DRotNet: a fully self-supervised approach to learn spatiotemporal features from unlabeled videos. A set of rotations are applied to all videos, and a pretext task is defined as prediction of these rotations. When accomplishing this task, 3DRotNet is actually trained to understand the semantic concepts and motions in videos. In other words, it learns a spatiotemporal video representation, which can be transferred to improve video understanding tasks in small datasets. Our extensive experiments successfully demonstrate the effectiveness of the proposed framework on action recognition, leading to significant improvements over the state-of-the-art self-supervised methods. With the self-supervised pre-trained 3DRotNet from large datasets, the recognition accuracy is boosted up by 20.4% on UCF101 and 16.7% on HMDB51 respectively, compared to the models trained from scratch.

Motivation & Objective

• Motivate learning video representations without labels due to labeling cost in large video datasets.
• Propose a rotation-prediction pretext task to jointly capture spatial and temporal video cues.
• Demonstrate that self-supervised pre-training improves downstream action recognition on small datasets.
• Show that features learned from rotation prediction transfer effectively across datasets and tasks.

Proposed method

• Use a 3D ResNet-18 backbone to learn spatiotemporal features.
• Apply four discrete video rotations (0°, 90°, 180°, 270°) to each video clip.
• Train F to predict the applied rotation via a classification objective (cross-entropy) or regression (optional).
• Pre-train on large unlabeled video datasets (Kinetics, Moment in Time) and transfer to action recognition tasks.
• Fine-tune the learned features on UCF101 and HMDB51 to evaluate transfer performance.

Experimental results

Research questions

• RQ1 Can a 3D CNN learn meaningful spatiotemporal video representations by solving a rotation-prediction pretext task without labels?
• RQ2 Do the self-supervised features transfer effectively to action recognition on small datasets compared to training from scratch and other self-supervised methods?
• RQ3 How do factors like rotation count, input modalities (RGB vs DIF), and clip length affect learned representations and transfer performance.

Key findings

• Pre-training on Kinetics/Moment in Time with rotation prediction yields significant gains over scratch: +20.4 percentage points on UCF101 and +16.7 on HMDB51 after transfer.
• The rotation-prediction task enables learning of both spatial and temporal cues, with attention maps showing foreground focus on moving subjects similar to supervised models.
• 3DRotNet-RGB, -DIF, and -Fusion variants outperform prior self-supervised methods on both UCF101 and HMDB51, with Fusion achieving 76.6% (UCF101) and 47.0% (HMDB51).
• Longer input clips and DIF (difference frames) provide stronger temporal signals, improving action recognition performance.
• More rotations used during pre-training (4 rotations) yield better transfer than fewer rotations; 8/360-degree setups did not outperform the 4-rotation configuration.