[Paper Review] Video Representation Learning by Dense Predictive Coding

Tengda Han, Weidi Xie+1 more|arXiv (Cornell University)|Sep 10, 2019

Human Pose and Action RecognitionComputer Science49 references48 citations

TL;DR

Dense Predictive Coding (DPC) learns self-supervised spatio-temporal video representations by predicting future embeddings in a dense, sequential manner, using a curriculum to extend future prediction and achieve strong action recognition performance from RGB frames alone.

ABSTRACT

The objective of this paper is self-supervised learning of spatio-temporal embeddings from video, suitable for human action recognition. We make three contributions: First, we introduce the Dense Predictive Coding (DPC) framework for self-supervised representation learning on videos. This learns a dense encoding of spatio-temporal blocks by recurrently predicting future representations; Second, we propose a curriculum training scheme to predict further into the future with progressively less temporal context. This encourages the model to only encode slowly varying spatial-temporal signals, therefore leading to semantic representations; Third, we evaluate the approach by first training the DPC model on the Kinetics-400 dataset with self-supervised learning, and then finetuning the representation on a downstream task, i.e. action recognition. With single stream (RGB only), DPC pretrained representations achieve state-of-the-art self-supervised performance on both UCF101(75.7% top1 acc) and HMDB51(35.7% top1 acc), outperforming all previous learning methods by a significant margin, and approaching the performance of a baseline pre-trained on ImageNet.

Motivation & Objective

• Motivate self-supervised learning for spatio-temporal video embeddings suitable for action recognition.
• Introduce Dense Predictive Coding (DPC) to predict future dense representations from past context.
• Propose a curriculum training scheme to predict further into the future with decreasing temporal context.
• Show that DPC with RGB streams achieves state-of-the-art self-supervised results on UCF101 and HMDB51 and approaches ImageNet-pretrained baselines.
• Evaluate the correlation between self-supervised gains and downstream supervised performance.

Proposed method

• Encode video blocks with a 3D-ResNet encoder to obtain z_t.
• Aggregate past latent codes z_t with a ConvGRU to form a context c_t.
• Predict future embeddings hat{z}_{t+1}, hat{z}_{t+2}, ... using a small predictor .
• Train with a dense, multi-way Noise Contrastive Estimation (NCE) loss across spatial locations and time steps.
• Utilize frame-wise augmentations to avoid reliance on optical flow and employ curriculum learning to extend future prediction horizons.
• Optionally fine-tune the learned representations on downstream action recognition tasks.

Experimental results

Research questions

• RQ1 Can self-supervised learning of dense spatio-temporal embeddings from RGB videos yield representations competitive for action recognition?
• RQ2 Does predicting into the future with a curriculum-style schedule improve semantic representation learning?
• RQ3 How does DPC compare to prior self-supervised methods on standard action recognition benchmarks?
• RQ4 Is a dense predictive, sequential approach essential for learning useful video representations compared to projection onto a single vector?

Key findings

• DPC with RGB streams achieves state-of-the-art self-supervised performance on UCF101 (top1 75.7% in some setups) and HMDB51 (top1 35.7%), outperforming prior RGB-only methods.
• Dense, sequential prediction of future spatio-temporal blocks with a Curriculum training scheme improves the learned representations and downstream action recognition.
• Pre-training on a larger dataset (Kinetics-400) yields stronger downstream performance than training on UCF101 alone, showing benefits from scale.
• There is a positive correlation between self-supervised accuracy during DPC pre-training and downstream supervised action recognition accuracy.
• Predicting further into the future (under curriculum) can enhance downstream performance even when self-supervised accuracy on the extended task is lower, indicating stronger semantic learning.