[Paper Review] Self-supervised Spatiotemporal Feature Learning by Video Geometric Transformations

Longlong Jing, Yingli Tian|arXiv (Cornell University)|Nov 28, 2018

Human Pose and Action RecognitionComputer Science35 references76 citations

TL;DR

This paper proposes a 3DConvNet-based self-supervised framework that learns spatiotemporal video features by using geometric transformations—such as 0°, 90°, 180°, and 270° rotations—as a pretext task, eliminating the need for human-annotated data. The method achieves state-of-the-art performance, boosting action recognition accuracy by 20.4% on UCF101 and 16.7% on HMDB51 compared to training from scratch, with 62.9% and 33.7% top-1 accuracy respectively.

ABSTRACT

To alleviate the expensive cost of data collection and annotation, many self-supervised learning methods were proposed to learn image representations without human-labeled annotations. However, self-supervised learning for video representations is not yet well-addressed. In this paper, we propose a novel 3DConvNet-based fully self-supervised framework to learn spatiotemporal video features without using any human-labeled annotations. First, a set of pre-designed geometric transformations (e.g. rotating 0 degree, 90 degrees, 180 degrees, and 270 degrees) are applied to each video. Then a pretext task can be defined as recognizing the pre-designed geometric transformations. Therefore, the spatiotemporal video features can be learned in the process of accomplishing this pretext task without using human-labeled annotations. The learned spatiotemporal video representations can further be employed as pretrained features for different video-related applications. The proposed geometric transformations (e.g. rotations) are proved to be effective to learn representative spatiotemporal features in our 3DConvNet-based fully self-supervised framework. With the pre-trained spatiotemporal features from two large video datasets, the performance of action recognition is significantly boosted up by 20.4% on UCF101 dataset and 16.7% on HMDB51 dataset respectively compared to that from the model trained from scratch. Furthermore, our framework outperforms the state-of-the-arts of fully self-supervised methods on both UCF101 and HMDB51 datasets and achieves 62.9% and 33.7% accuracy respectively.

Motivation & Objective

• To address the high cost of video data annotation by enabling self-supervised learning of spatiotemporal video features.
• To develop a fully self-supervised framework that eliminates reliance on human-labeled data for video representation learning.
• To improve action recognition performance using pre-trained features derived from geometric transformation pretext tasks.
• To demonstrate the effectiveness of geometric transformations as a supervisory signal for learning meaningful spatiotemporal features in videos.

Proposed method

• A set of predefined geometric transformations—0°, 90°, 180°, and 270° rotations—are applied to input video clips.
• A 3DConvNet is trained to predict the applied geometric transformation as a pretext task, learning spatiotemporal features in the process.
• The framework is trained end-to-end without any human-annotated labels, relying solely on the transformation prediction task.
• The learned features are fine-tuned for downstream video classification tasks such as action recognition.
• The method leverages spatial and temporal invariances induced by geometric transformations to learn robust video representations.
• The approach is evaluated on two large-scale video datasets to assess generalization and performance.

Experimental results

Research questions

• RQ1 Can geometric transformations serve as an effective supervisory signal for self-supervised video representation learning?
• RQ2 How well can a 3DConvNet learn spatiotemporal features through a rotation prediction pretext task without human annotations?
• RQ3 To what extent does pre-training with this method improve downstream action recognition performance compared to training from scratch?
• RQ4 How does this framework compare to state-of-the-art fully self-supervised video learning methods on standard benchmarks?

Key findings

• The proposed method achieves 62.9% top-1 accuracy on the UCF101 dataset, outperforming state-of-the-art fully self-supervised methods.
• On the HMDB51 dataset, the method attains 33.7% top-1 accuracy, setting a new SOTA for fully self-supervised video learning.
• Pre-training with the geometric transformation pretext task boosts action recognition accuracy by 20.4% on UCF101 compared to training from scratch.
• The method improves action recognition performance by 16.7% on HMDB51 relative to models trained without pre-training.
• Geometric transformations such as rotations are effective in learning representative spatiotemporal features in the absence of human annotations.
• The framework generalizes well across datasets, demonstrating the robustness of the self-supervised learning signal.