[Paper Review] ActionCLIP: A New Paradigm for Video Action Recognition

Mengmeng Wang, Jiazheng Xing+1 more|arXiv (Cornell University)|Sep 17, 2021

Human Pose and Action RecognitionComputer Science53 references189 citations

TL;DR

ActionCLIP reframes video action recognition as video–text matching, introduces a pre-train, prompt, and fine-tune paradigm, and achieves state-of-the-art results on Kinetics-400 with strong zero-shot/few-shot transfer.

ABSTRACT

The canonical approach to video action recognition dictates a neural model to do a classic and standard 1-of-N majority vote task. They are trained to predict a fixed set of predefined categories, limiting their transferable ability on new datasets with unseen concepts. In this paper, we provide a new perspective on action recognition by attaching importance to the semantic information of label texts rather than simply mapping them into numbers. Specifically, we model this task as a video-text matching problem within a multimodal learning framework, which strengthens the video representation with more semantic language supervision and enables our model to do zero-shot action recognition without any further labeled data or parameters requirements. Moreover, to handle the deficiency of label texts and make use of tremendous web data, we propose a new paradigm based on this multimodal learning framework for action recognition, which we dub "pre-train, prompt and fine-tune". This paradigm first learns powerful representations from pre-training on a large amount of web image-text or video-text data. Then it makes the action recognition task to act more like pre-training problems via prompt engineering. Finally, it end-to-end fine-tunes on target datasets to obtain strong performance. We give an instantiation of the new paradigm, ActionCLIP, which not only has superior and flexible zero-shot/few-shot transfer ability but also reaches a top performance on general action recognition task, achieving 83.8% top-1 accuracy on Kinetics-400 with a ViT-B/16 as the backbone. Code is available at https://github.com/sallymmx/ActionCLIP.git

Motivation & Objective

• Reframe action recognition as a video–text matching problem to leverage semantic label texts.
• Introduce a scalable paradigm—pre-train, prompt, and fine-tune—to reuse large web-data pre-trained models.
• Demonstrate zero-shot and few-shot transfer capabilities on standard benchmarks.
• Show that textual prompts and carefully designed visual prompts improve performance while avoiding catastrophic forgetting.

Proposed method

• Formulate action recognition as P(f(x,y)|θ) via a video encoder gV and a language encoder gW to maximize cross-modal similarity s(x,y).
• Use cosine similarity and symmetric p_x2y, p_y2x with a temperature τ to define a contrastive KL loss across video–text pairs.
• Adopt a multimodal training objective (video–text contrastive loss) to pull related video and label representations together.
• Propose a new paradigm: pre-train on web data, apply textual and visual prompts to align downstream tasks with pre-training objectives, then fine-tune end-to-end on target datasets.
• Instantiate ActionCLIP with CLIP as the base model, using textual prompts for label sentences and various visual prompts (pre-network, in-network, post-network) to model temporal information.
• Evaluate under zero-shot and few-shot conditions and compare against unimodal baselines and prior methods.

Experimental results

Research questions

• RQ1 Can semantic information in label texts improve action recognition compared to traditional unimodal classification?
• RQ2 Does a pre-train, prompt, and fine-tune paradigm enable effective zero-shot and few-shot action recognition on standard benchmarks?
• RQ3 How do textual prompts and different visual prompts affect performance and knowledge retention (catastrophic forgetting) when adapting pre-trained models to video actions?

Key findings

• Multimodal framework improves top-1 accuracy by 2.91 percentage points over unimodal baseline (75.45% to 78.36% on Kinetics-400 with the instantiation used).
• ActionCLIP achieves 82.6% top-1 and 96.2% top-5 on Kinetics-400 with ViT-B/16 at 16 frames and 82.6%/96.2% at 16 frames; reaches 83.8% top-1 with 32 frames (ViT-B/16).
• Zero-shot/few-shot results show ActionCLIP leading in data-poor settings and enabling zero-shot recognition on Kinetics-400, HMDB-51, and UCF-101 where some baselines struggle.
• Prompting the textual labels improves performance versus using only label words (77.82% -> 78.36% top-1).
• Visual prompts influence performance; post-network prompts (MeanP, LSTM, Conv1D, Transf) yield strong results while pre-network Joint and in-network Shift can reduce performance, indicating the importance of prompt design to prevent catastrophic forgetting.
• Fine-tuning all components yields the best results; freezing encoders degrades performance (e.g., V1–V4 comparisons in ablations).