[Paper Review] Weakly Supervised Temporal Action Localization via Representative Snippet Knowledge Propagation

Linjiang Huang, Liang Wang+1 more|arXiv (Cornell University)|Mar 6, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes a weakly supervised temporal action localization method that improves pseudo label quality by propagating knowledge from representative video snippets both within and across videos. Using a memory bank and a bipartite random walk module, it refines feature representations to generate more accurate temporal class activation maps, achieving state-of-the-art performance with a 1.2% mAP gain on THUMOS14 and consistent improvements across multiple baselines.

ABSTRACT

Weakly supervised temporal action localization aims to localize temporal boundaries of actions and simultaneously identify their categories with only video-level category labels. Many existing methods seek to generate pseudo labels for bridging the discrepancy between classification and localization, but usually only make use of limited contextual information for pseudo label generation. To alleviate this problem, we propose a representative snippet summarization and propagation framework. Our method seeks to mine the representative snippets in each video for propagating information between video snippets to generate better pseudo labels. For each video, its own representative snippets and the representative snippets from a memory bank are propagated to update the input features in an intra- and inter-video manner. The pseudo labels are generated from the temporal class activation maps of the updated features to rectify the predictions of the main branch. Our method obtains superior performance in comparison to the existing methods on two benchmarks, THUMOS14 and ActivityNet1.3, achieving gains as high as 1.2% in terms of average mAP on THUMOS14.

Motivation & Objective

• To address the discrepancy between video-level classification and snippet-level localization in weakly supervised temporal action localization.
• To improve pseudo label quality by leveraging contextual information beyond individual snippets.
• To enable intra- and inter-video knowledge transfer using representative snippets as knowledge carriers.
• To reduce reliance on discriminative or background snippets that mislead localization.
• To achieve state-of-the-art performance with minimal supervision, using only video-level labels.

Proposed method

• Identifies representative snippets per video using expectation-maximization attention to handle view, action, and background variations.
• Maintains a memory bank storing high-confidence representative snippets from all videos for inter-video knowledge transfer.
• Employs a bipartite random walk (BiRW) module to update video features by propagating knowledge from intra- and inter-video representative snippets.
• Uses the updated features to generate refined temporal class activation maps (TCAMs) as online pseudo labels for rectifying the main model's predictions.
• Integrates a dual-branch training scheme: the main branch learns from video-level labels, while an auxiliary branch learns from pseudo labels to improve focus on representative snippets.
• Applies residual connections to stabilize feature updates during iterative propagation, preventing gradient degradation.

Experimental results

Research questions

• RQ1 Can representative snippets from a video and a memory bank improve pseudo label quality in weakly supervised temporal action localization?
• RQ2 How does intra- and inter-video knowledge propagation from representative snippets affect localization accuracy?
• RQ3 Does propagating features rather than scores lead to better performance in weakly supervised settings?
• RQ4 Can the proposed framework consistently improve existing state-of-the-art methods with minimal architectural changes?
• RQ5 What is the optimal number of propagation iterations for balancing performance and training stability?

Key findings

• The proposed method achieves a 1.2% absolute gain in average mAP on THUMOS14 compared to prior methods, demonstrating state-of-the-art performance.
• On ActivityNet1.3, the method achieves superior performance, confirming its generalization across datasets.
• The ablation study shows that using weighted feature propagation (with residual connection) outperforms direct propagation or score-based propagation.
• The method consistently improves performance across multiple baselines, including STPN, BM, WUM, and FAC-Net, with gains of up to 4.8% mAP.
• The optimal number of propagation iterations is 5, as higher iterations lead to performance degradation due to gradient instability.
• The memory bank mechanism enables effective inter-video knowledge transfer while keeping GPU memory cost low.