[Paper Review] EZ-CLIP: Efficient Zeroshot Video Action Recognition

Shahzad Ahmad, Sukalpa Chanda+1 more|arXiv (Cornell University)|Dec 13, 2023

Human Pose and Action Recognition4 citations

TL;DR

EZ-CLIP proposes a computationally efficient adaptation of CLIP for zero-shot video action recognition using temporal visual prompting and a motion-aware loss, achieving state-of-the-art performance with only 5.2 million learnable parameters—trained on a single GPU—while preserving CLIP’s generalization capability and outperforming prior methods on multiple benchmarks.

ABSTRACT

Recent advancements in large-scale pre-training of visual-language models on paired image-text data have demonstrated impressive generalization capabilities for zero-shot tasks. Building on this success, efforts have been made to adapt these image-based visual-language models, such as CLIP, for videos extending their zero-shot capabilities to the video domain. While these adaptations have shown promising results, they come at a significant computational cost and struggle with effectively modeling the crucial temporal aspects inherent to the video domain. In this study, we present EZ-CLIP, a simple and efficient adaptation of CLIP that addresses these challenges. EZ-CLIP leverages temporal visual prompting for seamless temporal adaptation, requiring no fundamental alterations to the core CLIP architecture while preserving its remarkable generalization abilities. Moreover, we introduce a novel learning objective that guides the temporal visual prompts to focus on capturing motion, thereby enhancing its learning capabilities from video data. We conducted extensive experiments on five different benchmark datasets, thoroughly evaluating EZ-CLIP for zero-shot learning and base-to-novel video action recognition, and also demonstrating its potential for few-shot generalization.Impressively, with a mere 5.2 million learnable parameters (as opposed to the 71.1 million in the prior best model), EZ-CLIP can be efficiently trained on a single GPU, outperforming existing approaches in several evaluations.

Motivation & Objective

• Address the high computational cost and poor temporal modeling of existing CLIP adaptations for video action recognition.
• Preserve CLIP’s strong zero-shot generalization capability while enabling effective spatio-temporal representation learning.
• Develop a lightweight, parameter-efficient method to adapt image-based visual-language models to video tasks without fine-tuning the backbone.
• Enhance modeling of motion cues in videos through a novel motion constraint loss.
• Demonstrate robust performance in zero-shot, base-to-novel, and few-shot video action recognition across diverse datasets.

Proposed method

• Introduce temporal visual prompts—learnable embeddings injected into the input space of CLIP—to model temporal dependencies across video frames.
• Apply the prompts during inference by attending over frame-level features, enabling end-to-end learning of temporal dynamics with minimal parameters.
• Propose a motion loss that encourages the model to focus on motion-relevant regions by comparing feature maps across frames.
• Use a contrastive learning objective between video and text embeddings, with prompts guiding temporal attention without updating the original CLIP backbone.
• Train the model end-to-end with only the temporal visual prompts and motion loss, keeping the original CLIP vision and text encoders frozen.
• Leverage LLM-generated action descriptions as text prompts to improve generalization of text embeddings in zero-shot settings.

Experimental results

Research questions

• RQ1 Can a minimal, parameter-efficient adaptation of CLIP achieve strong zero-shot performance on video action recognition without fine-tuning the backbone?
• RQ2 How effectively can temporal visual prompting model long-range temporal dependencies in videos with minimal learnable parameters?
• RQ3 To what extent does a motion-aware loss improve the model’s ability to capture motion cues in videos compared to standard contrastive learning?
• RQ4 How does EZ-CLIP generalize in base-to-novel action recognition, especially on motion-intensive datasets like Something-Something-v2?
• RQ5 Can the model maintain high throughput and low FLOPs while outperforming existing CLIP-based video adaptation methods?

Key findings

• EZ-CLIP achieves state-of-the-art zero-shot accuracy on UCF-101 (95.9% top-1), HMDB-51 (79.8% top-1), and Something-Something-v2 (54.0% top-1) with only 5.2 million tunable parameters.
• On the challenging SSv2 dataset, EZ-CLIP achieves a 54.0% top-1 accuracy in base-to-novel evaluation, significantly outperforming prior methods with fewer parameters and higher throughput.
• The model demonstrates 322.4 throughput (TP) per view on a single A100 GPU, outperforming methods like ViFi-CLIP (71.1 TP) and ActionCLIP (67.7 TP) in efficiency.
• The motion loss leads to improved attention localization on motion regions: visualization shows EZ-CLIP focuses on moving objects, while the base CLIP model attends to background and appearance.
• LLM-generated action descriptions improve text embedding generalization, increasing accuracy by up to 2.5% on HMDB-51 and 11% on UCF-101 in zero-shot settings.
• t-SNE visualizations confirm that EZ-CLIP learns better-separated feature representations than the base model, especially for motion-dependent actions like 'putting something on surface'.