[Paper Review] Zero-Shot Temporal Action Detection via Vision-Language Prompting

Sauradip Nag, Xiatian Zhu+2 more|arXiv (Cornell University)|Jul 17, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes STALE, a novel one-stage zero-shot temporal action detection model that decouples localization and classification via parallel heads with a learnable, class-agnostic representation masking module, enabling end-to-end optimization and eliminating error propagation. STALE achieves state-of-the-art performance on ZS-TAD benchmarks, outperforming prior methods by large margins, and also surpasses strong supervised baselines on standard TAD datasets.

ABSTRACT

Existing temporal action detection (TAD) methods rely on large training data including segment-level annotations, limited to recognizing previously seen classes alone during inference. Collecting and annotating a large training set for each class of interest is costly and hence unscalable. Zero-shot TAD (ZS-TAD) resolves this obstacle by enabling a pre-trained model to recognize any unseen action classes. Meanwhile, ZS-TAD is also much more challenging with significantly less investigation. Inspired by the success of zero-shot image classification aided by vision-language (ViL) models such as CLIP, we aim to tackle the more complex TAD task. An intuitive method is to integrate an off-the-shelf proposal detector with CLIP style classification. However, due to the sequential localization (e.g, proposal generation) and classification design, it is prone to localization error propagation. To overcome this problem, in this paper we propose a novel zero-Shot Temporal Action detection model via Vision-LanguagE prompting (STALE). Such a novel design effectively eliminates the dependence between localization and classification by breaking the route for error propagation in-between. We further introduce an interaction mechanism between classification and localization for improved optimization. Extensive experiments on standard ZS-TAD video benchmarks show that our STALE significantly outperforms state-of-the-art alternatives. Besides, our model also yields superior results on supervised TAD over recent strong competitors. The PyTorch implementation of STALE is available at https://github.com/sauradip/STALE.

Motivation & Objective

• Address the challenge of zero-shot temporal action detection (ZS-TAD), where models must recognize unseen action classes without any labeled training data for those classes.
• Overcome the critical limitation of existing two-stage ZS-TAD methods that suffer from localization error propagation due to sequential processing of proposal generation and classification.
• Enable effective zero-shot transfer to unseen action classes by leveraging vision-language (ViL) models like CLIP through a novel prompting and adaptation mechanism.
• Improve cross-modal alignment between visual and textual representations in video understanding tasks to enhance generalization and performance under low-resource settings.
• Develop a unified, end-to-end trainable framework that jointly optimizes localization and classification, avoiding the incompatibility between frozen localization and classification heads in prior approaches.

Proposed method

• Propose a one-stage architecture with parallel localization and classification heads, decoupling the two tasks to prevent error propagation from localization to classification.
• Introduce a learnable, class-agnostic representation masking module that generates foreground masks from video features, enabling zero-shot generalization to unseen action classes.
• Utilize a vision-language model (CLIP-style) for zero-shot classification by prompting with natural language descriptions of action classes, leveraging pre-trained visual and textual encoders.
• Implement an inter-stream alignment regularization using self-attention in a Transformer-based framework to enhance cross-modal consistency between visual and language features.
• Design a learnable textual context with up to 50 tokens to better capture semantic context for action classification, improving performance over fixed or shorter prompts.
• Apply a multi-head Transformer (without positional encoding) for temporal modeling to capture long-range dependencies in video sequences, outperforming 1D-CNN and MS-TCN baselines.

Experimental results

Research questions

• RQ1 Can a one-stage, parallel architecture for temporal action detection eliminate error propagation between localization and classification in zero-shot settings?
• RQ2 How effective is a learnable, class-agnostic representation masking module in enabling zero-shot generalization to unseen action classes?
• RQ3 To what extent does cross-modal alignment between visual and language features improve zero-shot temporal action detection performance?
• RQ4 Does using a learnable textual context of variable length improve classification accuracy compared to fixed or shorter prompts?
• RQ5 Can the proposed STALE model achieve state-of-the-art performance not only in zero-shot but also in fully supervised temporal action detection settings?

Key findings

• STALE achieves a mean average precision (mAP) of 38.2% on the 75% seen split of ActivityNet, significantly outperforming prior SOTA methods on zero-shot temporal action detection.
• The ablation study shows that fine-tuning the text encoder improves mAP by 1.3% (from 36.9% to 38.2%), demonstrating the importance of domain adaptation for TAD tasks.
• Using a Transformer for temporal modeling yields 38.2% mAP, outperforming 1D-CNN (29.0%) and MS-TCN (33.5%) under the same zero-shot setting, confirming its superior contextual modeling ability.
• The inter-stream alignment regularization with vision-to-language cross-adaptation boosts mAP by 1.4% (from 23.5% to 24.9%), with minimal computational overhead.
• Increasing the number of context tokens from 10 to 50 improves mAP from 36.3% to 38.2%, indicating that longer context sequences enhance performance in ZS-TAD.
• STALE also achieves superior performance in the supervised TAD setting, outperforming recent strong competitors on ActivityNet and THUMOS benchmarks, demonstrating its generalization capability beyond zero-shot learning.