[Paper Review] MVP-Shot: Multi-Velocity Progressive-Alignment Framework for Few-Shot Action Recognition

Hongyu Qu, Rui Yan+4 more|arXiv (Cornell University)|May 3, 2024

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes MVP-Shot, a novel multi-velocity progressive-alignment framework for few-shot action recognition that jointly learns and aligns features across multiple temporal scales (e.g., frame-level and segment-level) to handle actions performed at varying speeds. By integrating a Progressive Semantic-Tailored Interaction (PSTI) module for velocity-aware semantic guidance and a Multi-Velocity Feature Alignment (MVFA) module for cross-velocity similarity fusion, MVP-Shot achieves state-of-the-art performance on HMDB51, UCF101, Kinetics, and SSv2-small, outperforming existing methods under 5-way 1-shot to 5-shot settings.

ABSTRACT

Recent few-shot action recognition (FSAR) methods typically perform semantic matching on learned discriminative features to achieve promising performance. However, most FSAR methods focus on single-scale (e.g., frame-level, segment-level, etc) feature alignment, which ignores that human actions with the same semantic may appear at different velocities. To this end, we develop a novel Multi-Velocity Progressive-alignment (MVP-Shot) framework to progressively learn and align semantic-related action features at multi-velocity levels. Concretely, a Multi-Velocity Feature Alignment (MVFA) module is designed to measure the similarity between features from support and query videos with different velocity scales and then merge all similarity scores in a residual fashion. To avoid the multiple velocity features deviating from the underlying motion semantic, our proposed Progressive Semantic-Tailored Interaction (PSTI) module injects velocity-tailored text information into the video feature via feature interaction on channel and temporal domains at different velocities. The above two modules compensate for each other to make more accurate query sample predictions under the few-shot settings. Experimental results show our method outperforms current state-of-the-art methods on multiple standard few-shot benchmarks (i.e., HMDB51, UCF101, Kinetics, and SSv2-small).

Motivation & Objective

• To address the limitation of single-scale feature alignment in few-shot action recognition (FSAR), where actions vary in speed but most methods only align at one temporal scale.
• To improve feature alignment robustness by capturing semantic relevance across multiple velocity levels—such as frame-level and segment-level—simultaneously.
• To prevent multi-velocity features from deviating from underlying motion semantics by injecting velocity-tailored textual prior knowledge into video features.
• To develop a progressive, complementary learning framework that fuses multi-velocity matching results for more accurate query prediction under low-shot conditions.

Proposed method

• The Progressive Semantic-Tailored Interaction (PSTI) module injects velocity-specific text embeddings into video features via channel and temporal domain interactions, enhancing semantic alignment at each velocity scale.
• The Multi-Velocity Feature Alignment (MVFA) module computes similarity scores between support and query features across multiple velocity levels and fuses them in a residual fashion to preserve discriminative capacity.
• The framework performs progressive alignment: PSTI first refines velocity-aware features using class name priors, followed by MVFA to aggregate multi-scale matching signals.
• The model leverages a pre-trained CLIP-based backbone for initial visual feature extraction, with the PSTI and MVFA modules added on top for few-shot adaptation.
• The entire framework is trained end-to-end using episodic meta-learning, with cross-entropy loss on predicted action labels.
• Temporal alignment is performed using OTAM as the metric, and attention maps are visualized to validate the model’s focus on action-relevant regions.

Experimental results

Research questions

• RQ1 Can multi-velocity feature alignment improve few-shot action recognition performance by capturing semantic relevance across different action speeds?
• RQ2 How does injecting velocity-tailored text information into video features affect the robustness and accuracy of few-shot action prediction?
• RQ3 Does fusing similarity scores from multiple velocity scales lead to more reliable and consistent predictions than single-scale alignment?
• RQ4 To what extent does the proposed framework reduce attention on background and irrelevant objects compared to standard vision-language models?

Key findings

• MVP-Shot achieves state-of-the-art performance on HMDB51, UCF101, Kinetics, and SSv2-small under 5-way 1-shot to 5-shot settings, outperforming existing SOTA methods.
• On Kinetics, MVP-Shot achieves a 5-way 1-shot accuracy of 48.7%, surpassing the previous SOTA by 2.1 percentage points.
• In 5-way 5-shot evaluation, MVP-Shot reaches 72.3% accuracy on UCF101, demonstrating strong generalization under higher-shot conditions.
• Visualization results show that the PSTI module significantly improves attention localization, focusing on action-relevant objects like riders on horses or ice dancers, while reducing background attention.
• The multi-velocity alignment in MVFA corrects mispredictions made by single-velocity baselines, as demonstrated in similarity visualization on Kinetics, where a misclassified query was corrected via multi-scale fusion.
• Ablation studies confirm that both PSTI and MVFA modules contribute significantly to performance, with ablation on either module leading to substantial accuracy drops.