[Paper Review] Fine-grained Temporal Contrastive Learning for Weakly-supervised Temporal Action Localization

Junyu Gao, Mengyuan Chen+1 more|arXiv (Cornell University)|Mar 31, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes Fine-grained Temporal Contrastive Learning (FTCL), a novel framework for weakly-supervised temporal action localization that leverages differentiable dynamic programming to learn fine-grained temporal distinctions between video sequences. By introducing two complementary contrastive objectives—Fine-grained Sequence Distance (FSD) and Longest Common Subsequence (LCS) contrasting—FTCL enhances action-background separation and reduces the task gap between classification and localization, achieving state-of-the-art performance on THUMOS14 and ActivityNet-1.2 benchmarks.

ABSTRACT

We target at the task of weakly-supervised action localization (WSAL), where only video-level action labels are available during model training. Despite the recent progress, existing methods mainly embrace a localization-by-classification paradigm and overlook the fruitful fine-grained temporal distinctions between video sequences, thus suffering from severe ambiguity in classification learning and classification-to-localization adaption. This paper argues that learning by contextually comparing sequence-to-sequence distinctions offers an essential inductive bias in WSAL and helps identify coherent action instances. Specifically, under a differentiable dynamic programming formulation, two complementary contrastive objectives are designed, including Fine-grained Sequence Distance (FSD) contrasting and Longest Common Subsequence (LCS) contrasting, where the first one considers the relations of various action/background proposals by using match, insert, and delete operators and the second one mines the longest common subsequences between two videos. Both contrasting modules can enhance each other and jointly enjoy the merits of discriminative action-background separation and alleviated task gap between classification and localization. Extensive experiments show that our method achieves state-of-the-art performance on two popular benchmarks. Our code is available at https://github.com/MengyuanChen21/CVPR2022-FTCL.

Motivation & Objective

• To address the severe ambiguity in weakly-supervised temporal action localization (WSAL) caused by lack of frame-wise annotations.
• To improve discriminative action-background separation and reduce the task gap between video-level classification and temporal localization.
• To exploit fine-grained temporal distinctions between videos as an inductive bias for better representation learning.
• To design a model-agnostic, non-intrusive contrastive learning framework that enhances existing WSAL methods without increasing inference cost.

Proposed method

• Proposes a differentiable dynamic programming formulation to enable end-to-end learning of temporal sequence relations.
• Introduces Fine-grained Sequence Distance (FSD) contrasting that models snippet-level relations using match, insert, and delete operations.
• Designs Longest Common Subsequence (LCS) contrasting to mine the longest common temporal substructures between two videos.
• Combines FSD and LCS objectives to jointly optimize for discriminative action-background separation and improved classification-to-localization transfer.
• Employs a contrastive learning strategy that leverages structural video relations without requiring trimmed videos or additional supervision.
• Integrates the proposed modules into existing WSAL frameworks as a plug-in component with no additional inference cost.

Experimental results

Research questions

• RQ1 Can fine-grained temporal distinctions between video sequences serve as an effective inductive bias for weakly-supervised action localization?
• RQ2 How can we effectively model both action-background and action-to-action temporal relations in a weakly-supervised setting?
• RQ3 Can differentiable dynamic programming be used to learn sequence-to-sequence relations without requiring aligned or trimmed videos?
• RQ4 Does combining FSD and LCS contrasting improve both action-background separation and localization accuracy compared to individual objectives?
• RQ5 Can the proposed contrastive learning framework be universally applied to enhance existing WSAL models?

Key findings

• FTCL achieves state-of-the-art performance on THUMOS14, with a mean average precision (mAP@t-IoU) of 43.6% across all IoU thresholds.
• On ActivityNet-1.2, the proposed method achieves a mAP@t-IoU of 43.6%, outperforming existing SOTA methods.
• Abllation studies confirm that both FSD and LCS contrasting are effective and complementary, with the full model outperforming variants that exclude either component.
• Replacing FSD and LCS with standard differential DTW yields inferior results, validating the superiority of the proposed contrastive objectives.
• The method consistently improves three strong baselines—STPN, W-TALC, and CoLA—by up to +1.9% in mAP, demonstrating its generalization and complementary nature.
• The framework is model-agnostic and incurs no additional computational cost during inference, making it practical for real-world deployment.