[Paper Review] Few-shot Action Recognition via Intra- and Inter-Video Information Maximization

Huabin Liu, Weiyao Lin+4 more|arXiv (Cornell University)|May 10, 2023

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes Video Information Maximization (VIM), a novel few-shot action recognition framework that enhances performance by jointly maximizing intra-video and inter-video information. VIM uses an adaptive spatial-temporal video sampler to select key frames and amplify discriminative regions, while a spatial-temporal action alignment model improves inter-video similarity measurement via feature-level alignment, achieving state-of-the-art results on UCF101, SSv2, and HMDB51 with mutual information-based auxiliary losses.

ABSTRACT

Current few-shot action recognition involves two primary sources of information for classification:(1) intra-video information, determined by frame content within a single video clip, and (2) inter-video information, measured by relationships (e.g., feature similarity) among videos. However, existing methods inadequately exploit these two information sources. In terms of intra-video information, current sampling operations for input videos may omit critical action information, reducing the utilization efficiency of video data. For the inter-video information, the action misalignment among videos makes it challenging to calculate precise relationships. Moreover, how to jointly consider both inter- and intra-video information remains under-explored for few-shot action recognition. To this end, we propose a novel framework, Video Information Maximization (VIM), for few-shot video action recognition. VIM is equipped with an adaptive spatial-temporal video sampler and a spatiotemporal action alignment model to maximize intra- and inter-video information, respectively. The video sampler adaptively selects important frames and amplifies critical spatial regions for each input video based on the task at hand. This preserves and emphasizes informative parts of video clips while eliminating interference at the data level. The alignment model performs temporal and spatial action alignment sequentially at the feature level, leading to more precise measurements of inter-video similarity. Finally, These goals are facilitated by incorporating additional loss terms based on mutual information measurement. Consequently, VIM acts to maximize the distinctiveness of video information from limited video data. Extensive experimental results on public datasets for few-shot action recognition demonstrate the effectiveness and benefits of our framework.

Motivation & Objective

• To address the underutilization of intra-video and inter-video information in few-shot action recognition.
• To improve recognition performance with limited labeled video data by maximizing informative content at both video and inter-video levels.
• To overcome challenges in frame sampling and action misalignment that degrade feature quality and similarity measurement.
• To develop a unified framework that jointly optimizes intra- and inter-video information for improved few-shot generalization.
• To validate the effectiveness of mutual information-based auxiliary losses in enhancing video representation learning.

Proposed method

• Introduces an adaptive spatial-temporal video sampler that dynamically selects key frames and amplifies discriminative spatial regions based on task-specific importance.
• Employs a two-stage action alignment model: temporal alignment (TC) to filter irrelevant frames and spatial alignment (SC) to focus on action-specific regions across videos.
• Uses mutual information maximization as auxiliary loss to guide both intra-video (frame-level) and inter-video (video-level) representation learning.
• Applies feature-level alignment through a contrastive learning objective that encourages similarity between aligned support and query video features.
• Integrates the video sampler and alignment model in an end-to-end trainable framework to jointly optimize data-level and feature-level information maximization.
• Employs a ProtoNet-based classification head with learned prototypes for few-shot classification, enhanced by the refined features from the VIM pipeline.

Figure 1 : Overall pipeline of VIM. The few-shot action recognition involves two primary sources of information: (1) Intra-video information and (2) Inter-video information. Our proposed VIM newly introduces an adaptive sampling and action alignment, which seeks to maximize the inter- and intra-vide

Experimental results

Research questions

• RQ1 Can adaptive sampling improve the utilization of intra-video information in few-shot action recognition?
• RQ2 Can spatial-temporal action alignment reduce the impact of action misalignment and improve inter-video similarity measurement?
• RQ3 Does jointly maximizing intra- and inter-video information lead to better few-shot generalization?
• RQ4 How effective are mutual information-based auxiliary losses in guiding representation learning for few-shot video recognition?
• RQ5 To what extent does VIM generalize across domains without fine-tuning?

Key findings

• VIM achieves state-of-the-art performance on UCF101 and SSv2 datasets under few-shot settings, outperforming existing methods including TRX, OTAM, and HyRSM.
• On 5-way 1-shot settings, VIM achieves 85.6% accuracy on UCF101 and 68.4% on SSv2, demonstrating strong few-shot generalization.
• The ablation study confirms that both the adaptive video sampler and action alignment module significantly improve performance, with mutual information losses contributing to 3.2% and 2.8% gains respectively.
• VIM demonstrates strong domain generalization: Kinetics-pretrained VIM generalizes well to UCF and HMDB, achieving comparable performance to standard training on these datasets.
• The inference speed of VIM is faster than OTAM and TRX, and it has lower parameter count than HyRSM, making it efficient despite 32.4M additional parameters.
• Visualization results show that the sampler preserves key action frames and highlights discriminative regions, while the alignment model effectively aligns temporal and spatial action evolutions across videos.

Figure 2 : (a) Uniform sampler may overlook frames containing key actions. Critical regions involving the actors and objects may be too small to be properly recognized. (b) Our sampler is able to (I) select frames from an entire video that contribute most to recognition, (II) amplify discriminative