[Paper Review] Exploring Temporal Granularity in Self-Supervised Video Representation Learning

Rui Qian, Yeqing Li+8 more|arXiv (Cornell University)|Dec 8, 2021

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes TeG, a self-supervised video representation learning framework that jointly optimizes for both fine-grained and persistent temporal features by contrasting dense temporal embeddings within a long-short clip pair and their global embeddings. TeG achieves state-of-the-art performance on 8 video benchmarks, outperforming supervised pre-training on most tasks by leveraging task-specific temporal granularity.

ABSTRACT

This work presents a self-supervised learning framework named TeG to explore Temporal Granularity in learning video representations. In TeG, we sample a long clip from a video and a short clip that lies inside the long clip. We then extract their dense temporal embeddings. The training objective consists of two parts: a fine-grained temporal learning objective to maximize the similarity between corresponding temporal embeddings in the short clip and the long clip, and a persistent temporal learning objective to pull together global embeddings of the two clips. Our study reveals the impact of temporal granularity with three major findings. 1) Different video tasks may require features of different temporal granularities. 2) Intriguingly, some tasks that are widely considered to require temporal awareness can actually be well addressed by temporally persistent features. 3) The flexibility of TeG gives rise to state-of-the-art results on 8 video benchmarks, outperforming supervised pre-training in most cases.

Motivation & Objective

• To investigate how different video tasks require features of varying temporal granularities.
• To address the limitation of existing self-supervised video methods that prioritize only temporal persistence, often underperforming on fine-grained temporal tasks.
• To develop a flexible framework that can learn both fine-grained and persistent representations simultaneously.
• To demonstrate that temporally persistent features can be sufficient for some tasks previously thought to require fine-grained awareness.
• To achieve state-of-the-art results across diverse video benchmarks by adapting to task-specific temporal granularity.

Proposed method

• Sample a long clip and a short clip that lies entirely within the long clip from a video.
• Feed both clips through a video encoder without final temporal average pooling to preserve temporal resolution.
• Project the encoded features into two separate embedding spaces: one for fine-grained temporal learning and one for persistent global learning.
• Apply a dense contrastive loss between corresponding temporal embeddings in the short and long clips to encourage fine-grained discriminability.
• Apply a global contrastive loss between the pooled global embeddings of the two clips to enforce temporal persistence.
• Balance the two objectives using a learnable loss weight α, allowing flexibility in prioritizing either granularity based on downstream task needs.

Experimental results

Research questions

• RQ1 Do different video understanding tasks require features of different temporal granularities?
• RQ2 Can temporally persistent features alone achieve strong performance on tasks traditionally considered to require fine-grained temporal awareness?
• RQ3 Can a unified self-supervised framework effectively learn both fine-grained and persistent representations without task-specific retraining?
• RQ4 How does the sampling strategy and temporal aggregation design affect the quality of learned features?
• RQ5 What is the optimal trade-off between fine-grained and persistent learning objectives for various downstream tasks?

Key findings

• Different video tasks require features of different temporal granularities: fine-grained features improve performance on VidSitu event classification by 2.8% and Kinetics-GEBD by 2.4%, while persistent features are superior for video-level recognition and spatiotemporal localization.
• Some tasks previously assumed to require fine-grained features, such as video-level action recognition on Kinetics, actually perform better with temporally persistent features, challenging prior assumptions.
• TeG-PS (persistent-only) outperforms supervised pre-training on AVA-Kinetics by 8.9 mAP using the same R3D-50 backbone, demonstrating the strength of persistent learning.
• TeG-FG (fine-grained-only) achieves 27.7 mAP on Kinetics and 28.7 mAP on AVA-Kinetics, surpassing prior unsupervised methods and showing the effectiveness of fine-grained learning.
• The ablation study confirms that loss weight α must be carefully tuned: setting α = 0.9 balances both objectives, as α = 1.0 (fully fine-grained) harms performance on persistent tasks.
• The proposed sampling strategy—long clip and contained short clip—yields the best performance (31.1 on VidSitu), outperforming symmetric short-short sampling (27.4) and random sampling (26.9).