[Paper Review] Motion-Focused Contrastive Learning of Video Representations

Rui Li, Yiheng Zhang+4 more|arXiv (Cornell University)|Jan 11, 2022

Human Pose and Action Recognition23 references4 citations

TL;DR

This paper proposes Motion-Focused Contrastive Learning (MCL), a self-supervised video representation learning method that leverages optical flow to guide both data augmentation and feature learning. By sampling tubelets based on spatio-temporal motion maps and aligning convolutional layer gradients to motion maps, MCL achieves state-of-the-art performance, outperforming ImageNet-supervised pre-training on UCF101 (81.91% top-1 accuracy) and Kinetics-400 (66.62% top-1 accuracy) under linear evaluation protocols.

ABSTRACT

Motion, as the most distinct phenomenon in a video to involve the changes over time, has been unique and critical to the development of video representation learning. In this paper, we ask the question: how important is the motion particularly for self-supervised video representation learning. To this end, we compose a duet of exploiting the motion for data augmentation and feature learning in the regime of contrastive learning. Specifically, we present a Motion-focused Contrastive Learning (MCL) method that regards such duet as the foundation. On one hand, MCL capitalizes on optical flow of each frame in a video to temporally and spatially sample the tubelets (i.e., sequences of associated frame patches across time) as data augmentations. On the other hand, MCL further aligns gradient maps of the convolutional layers to optical flow maps from spatial, temporal and spatio-temporal perspectives, in order to ground motion information in feature learning. Extensive experiments conducted on R(2+1)D backbone demonstrate the effectiveness of our MCL. On UCF101, the linear classifier trained on the representations learnt by MCL achieves 81.91% top-1 accuracy, outperforming ImageNet supervised pre-training by 6.78%. On Kinetics-400, MCL achieves 66.62% top-1 accuracy under the linear protocol. Code is available at https://github.com/YihengZhang-CV/MCL-Motion-Focused-Contrastive-Learning.

Motivation & Objective

• To investigate the importance of motion in self-supervised video representation learning.
• To improve contrastive learning by explicitly incorporating motion information into data augmentation and feature optimization.
• To develop a method that samples meaningful spatio-temporal tubelets based on motion dynamics for better contrastive learning.
• To align gradient maps of convolutional layers with optical flow maps to ground motion in learned features.
• To demonstrate that motion-focused self-supervised pre-training can surpass ImageNet-supervised pre-training on downstream video benchmarks.

Proposed method

• Extract dense optical flow for each frame using the unsupervised TV-L1 algorithm to generate motion maps.
• Construct a spatio-temporal motion map (ST-motion) for each video clip by stacking sequential frame-level motion maps.
• Use 3D average pooling on ST-motion to compute clip-level motion and select high-motion clips as candidates for temporal augmentation.
• Apply temporal pooling on ST-motion maps of selected clips to estimate spatial motion (S-motion) and localize temporally consistent spatial patches as tubelets.
• For feature learning, extract ST-motion maps for each tubelet and apply spatial or temporal pooling to generate T-motion or S-motion maps.
• Minimize mean squared error between gradient maps of convolutional layers and motion maps (S-motion, T-motion, ST-motion) to align features with motion dynamics.

Experimental results

Research questions

• RQ1 How critical is motion for self-supervised video representation learning?
• RQ2 Can motion-guided data augmentation improve contrastive learning in video representation learning?
• RQ3 Can aligning feature gradients with motion maps enhance the discriminative power of learned representations?
• RQ4 Does motion-focused self-supervised pre-training surpass ImageNet-supervised pre-training on video benchmarks?
• RQ5 How transferable are representations learned via MCL to downstream video understanding tasks?

Key findings

• On UCF101, MCL achieves 81.91% top-1 accuracy under the linear evaluation protocol, outperforming ImageNet-supervised pre-training by 6.78 percentage points.
• On Kinetics-400, MCL achieves 66.62% top-1 accuracy under the linear protocol, demonstrating strong generalization on larger-scale datasets.
• When fine-tuned on UCF101 and HMDB51, MCL pre-training with R(2+1)D backbone achieves 90.40% and 61.30% accuracy, respectively, surpassing prior SOTA methods including SeCo and DynamoNet.
• On video retrieval, MCL with R(2+1)D backbone achieves a Recall@1 score of 80.4% on UCF101 and 68.9% on HMDB51, outperforming CoCLR with two-stream inputs.
• Grad-CAM visualization shows that MCL's attention maps align more closely with S-motion maps than SeCo, indicating better localization of motion-relevant regions.
• The method demonstrates strong transferability, achieving state-of-the-art results across multiple downstream tasks including action recognition and video retrieval.