[Paper Review] Self-supervised Co-training for Video Representation Learning

Tengda Han, Weidi Xie+1 more|arXiv (Cornell University)|Oct 19, 2020

Human Pose and Action Recognition72 references273 citations

TL;DR

The paper introduces CoCLR, a self-supervised co-training framework that exchanges positives across RGB and optical-flow views to improve contrastive learning, achieving near-UberNCE performance with greater training efficiency on video action recognition and retrieval.

ABSTRACT

The objective of this paper is visual-only self-supervised video representation learning. We make the following contributions: (i) we investigate the benefit of adding semantic-class positives to instance-based Info Noise Contrastive Estimation (InfoNCE) training, showing that this form of supervised contrastive learning leads to a clear improvement in performance; (ii) we propose a novel self-supervised co-training scheme to improve the popular infoNCE loss, exploiting the complementary information from different views, RGB streams and optical flow, of the same data source by using one view to obtain positive class samples for the other; (iii) we thoroughly evaluate the quality of the learnt representation on two different downstream tasks: action recognition and video retrieval. In both cases, the proposed approach demonstrates state-of-the-art or comparable performance with other self-supervised approaches, whilst being significantly more efficient to train, i.e. requiring far less training data to achieve similar performance.

Motivation & Objective

• Investigate whether instance-discrimination alone best utilizes video data for self-supervised learning.
• Assess whether hard positives from semantic classes can improve contrastive video representations.
• Propose a self-supervised co-training scheme (CoCLR) that mines positives across complementary views (RGB and flow).
• Evaluate the learned representations on downstream tasks: action recognition and video retrieval on UCF101, HMDB51, and Kinetics-400.

Proposed method

• Compare InfoNCE baseline (instance discrimination) with an oracle UberNCE using semantic labels.
• Introduce CoCLR to mine cross-view positives: use top-K similar clips from the flow view to augment RGB training, and vice versa.
• Alternate optimization between RGB and flow networks to progressively improve representations.
• Use a two-stage training: (i) independent InfoNCE pretraining for RGB and flow, (ii) alternating co-training using cross-view positives.
• Evaluate with a linear probe and retrieval to measure transferability of the learned representations.

Experimental results

Research questions

• RQ1 Does incorporating semantic-class positives (UberNCE) improve over instance-only InfoNCE in video representation learning?
• RQ2 Can co-training across RGB and optical-flow views harvest harder positives and improve downstream performance?
• RQ3 How does CoCLR compare to single-view self-supervised methods and to UberNCE in action recognition and retrieval?
• RQ4 What are the effects of hyperparameters such as top-K positive mining (K) and alternation cycles on CoCLR performance?

Key findings

• UberNCE outperforms InfoNCE, illustrating that instance discrimination may waste data resources.
• CoCLR significantly improves over InfoNCE and CMC, approaching UberNCE performance in linear-probe action recognition (RGB) and retrieval.
• Two-stream CoCLR (RGB+Flow) further improves results, with the RGB and Flow models providing complementary gains.
• End-to-end finetuning reduces performance gaps between training schemes, but CoCLR remains superior in pretraining transfer scenarios.
• CoCLR demonstrates state-of-the-art or comparable results to other self-supervised methods on UCF101 and Kinetics-400, with higher training efficiency and less data required.