[Paper Review] Pretext-Contrastive Learning: Toward Good Practices in Self-supervised Video Representation Leaning

Tao Li, Xueting Wang+1 more|arXiv (Cornell University)|Oct 29, 2020

Human Pose and Action Recognition49 references8 citations

TL;DR

This paper proposes Pretext-Contrastive Learning (PCL), a joint optimization framework that combines pretext tasks and contrastive learning with improved data augmentation and preprocessing to significantly boost self-supervised video representation learning. With extensive ablation and evaluation, PCL achieves state-of-the-art performance on UCF101 and HMDB51 using only 3.6% of Kinetics data, setting new benchmarks for self-supervised video learning.

ABSTRACT

Recently, pretext-task based methods are proposed one after another in self-supervised video feature learning. Meanwhile, contrastive learning methods also yield good performance. Usually, new methods can beat previous ones as claimed that they could capture "better" temporal information. However, there exist setting differences among them and it is hard to conclude which is better. It would be much more convincing in comparison if these methods have reached as closer to their performance limits as possible. In this paper, we start from one pretext-task baseline, exploring how far it can go by combining it with contrastive learning, data pre-processing, and data augmentation. A proper setting has been found from extensive experiments, with which huge improvements over the baselines can be achieved, indicating a joint optimization framework can boost both pretext task and contrastive learning. We denote the joint optimization framework as Pretext-Contrastive Learning (PCL). The other two pretext task baselines are used to validate the effectiveness of PCL. And we can easily outperform current state-of-the-art methods in the same training manner, showing the effectiveness and the generality of our proposal. It is convenient to treat PCL as a standard training strategy and apply it to many other works in self-supervised video feature learning.

Motivation & Objective

• To investigate how pretext tasks and contrastive learning can be jointly optimized to improve self-supervised video representation learning.
• To identify optimal training settings—especially data augmentation, preprocessing, and hyperparameters—that maximize performance of pretext-based methods.
• To establish a generalizable, flexible training strategy that outperforms existing self-supervised methods without requiring new pretext tasks or contrastive objectives.
• To set new performance baselines on standard video benchmarks (UCF101 and HMDB51) under consistent training protocols.
• To validate the generality and effectiveness of the proposed framework across multiple pretext task baselines and network architectures.

Proposed method

• Proposes a joint optimization framework, Pretext-Contrastive Learning (PCL), that simultaneously trains a pretext task head and a contrastive learning head on the same video features.
• Integrates strong data augmentation strategies, including random cropping, color jittering, and temporal cropping, to improve feature robustness.
• Applies residual clips (clips with skip-connections) to enhance feature learning by preserving long-range temporal dependencies.
• Uses a dual-head training setup: one head for the pretext task (e.g., video clip order prediction), and another for contrastive learning using positive/negative sample pairs.
• Employs a symmetric contrastive loss (e.g., InfoNCE) to encourage feature discrimination between different views of the same video.
• Optimizes both heads end-to-end with a weighted sum of their respective losses, enabling mutual improvement.

Experimental results

Research questions

• RQ1 Can combining pretext tasks and contrastive learning lead to better video representation learning than using either method alone?
• RQ2 What specific data augmentation and preprocessing techniques maximize performance when jointly training pretext and contrastive objectives?
• RQ3 Does the joint framework PCL generalize across different pretext task baselines and network architectures?
• RQ4 Can PCL achieve state-of-the-art performance with significantly less pre-training data than current methods?
• RQ5 What are the optimal hyperparameters and training settings that unlock the full potential of joint pretext-contrastive learning?

Key findings

• PCL achieves a 66.0% action recognition accuracy on UCF101 using only 3.6% of Kinetics data, significantly outperforming previous self-supervised methods under the same training regime.
• On HMDB51, PCL achieves 54.1% top-1 accuracy, setting a new state-of-the-art for self-supervised video learning with minimal data.
• In video retrieval, PCL outperforms both standalone pretext and contrastive learning baselines across 72 out of 101 classes on UCF101, with superior average performance.
• Ablation studies confirm that strong data augmentation and residual clips are critical components, contributing significantly to performance gains.
• The framework generalizes well: PCL improves performance across three different pretext task baselines (VCP, etc.) and multiple network backbones (R3D, ResNeXt).
• Visualization shows that PCL features are more clustered and discriminative than those from baseline methods, indicating better feature quality.