[Paper Review] Online Learnable Keyframe Extraction in Videos and its Application with Semantic Word Vector in Action Recognition

G M Mashrur E Elahi, Yee‐Hong Yang|arXiv (Cornell University)|Sep 25, 2020

Video Analysis and Summarization37 references4 citations

TL;DR

This paper proposes an online learnable keyframe extraction module (OKFEM) that selects discriminative video frames in real time using a learnable threshold and motion-based features, significantly reducing computational load. The method achieves state-of-the-art performance in video summarization and action recognition by using only 30% of frames, with further gains when combined with semantic word vectors via a novel plugin module.

ABSTRACT

Video processing has become a popular research direction in computer vision due to its various applications such as video summarization, action recognition, etc. Recently, deep learning-based methods have achieved impressive results in action recognition. However, these methods need to process a full video sequence to recognize the action, even though most of these frames are similar and non-essential to recognizing a particular action. Additionally, these non-essential frames increase the computational cost and can confuse a method in action recognition. Instead, the important frames called keyframes not only are helpful in the recognition of an action but also can reduce the processing time of each video sequence for classification or in other applications, e.g. summarization. As well, current methods in video processing have not yet been demonstrated in an online fashion. Motivated by the above, we propose an online learnable module for keyframe extraction. This module can be used to select key-shots in video and thus can be applied to video summarization. The extracted keyframes can be used as input to any deep learning-based classification model to recognize action. We also propose a plugin module to use the semantic word vector as input along with keyframes and a novel train/test strategy for the classification models. To our best knowledge, this is the first time such an online module and train/test strategy have been proposed. The experimental results on many commonly used datasets in video summarization and in action recognition have shown impressive results using the proposed module.

Motivation & Objective

• To address the high computational cost and redundancy in full-video deep learning methods for action recognition.
• To overcome the limitations of existing keyframe extraction methods, particularly their offline nature and reliance on manual or complex frame selection.
• To develop an online, end-to-end trainable module that selects keyframes sequentially using only immediate neighboring frame information.
• To enhance action recognition performance by integrating semantic word vectors as a supplementary input modality.
• To establish a new training and inference strategy (ITTS) that improves model generalization and accuracy in keyframe-based recognition.

Proposed method

• Proposes an online keyframe extraction module (OKFEM) that processes video frames sequentially, using motion features and a learnable threshold (TH) to identify keyframes.
• Leverages deformable convolutional features to capture spatiotemporal dynamics and guide keyframe selection based on local motion changes.
• Introduces a learnable threshold mechanism that adapts during training to optimize keyframe selection based on frame dissimilarity and motion energy.
• Designs a plugin module that fuses semantic word vectors (W2V) with keyframe features to enrich semantic context for action classification.
• Develops a novel train/test strategy (ITTS) that enables effective fine-tuning and inference using only keyframes and word vectors.
• Employs a hybrid loss function combining cross-entropy and frame-level supervision to improve keyframe selection accuracy.

Experimental results

Research questions

• RQ1 Can an online, learnable keyframe extraction method reduce computational cost while maintaining or improving action recognition accuracy?
• RQ2 How effective is the proposed OKFEM module in selecting representative frames compared to full-video processing?
• RQ3 To what extent does integrating semantic word vectors with keyframes improve action recognition performance?
• RQ4 Can the proposed ITTS training strategy enhance model generalization in keyframe-based recognition?
• RQ5 Does the method achieve state-of-the-art performance in video summarization and action recognition benchmarks?

Key findings

• OKFEM reduces frame usage to less than 30% of the original video while maintaining high recognition accuracy, significantly lowering computational cost.
• On the HMDB51 dataset, OKFEM+I3D outperforms the original I3D model by 3.6% accuracy, demonstrating the effectiveness of keyframe-based representation.
• The addition of the W2V plugin module boosts performance, with OKFEM+W2V+I3D achieving 87.9% accuracy on HMDB51, surpassing the baseline by 4.1%.
• On Something-Something-V1 and Kinetics-400, OKFEM+W2V+ResNet152 achieves state-of-the-art results, outperforming models using full video sequences.
• Qualitative analysis confirms that selected keyframes closely align with ground-truth importance scores, validating the method's precision in frame selection.
• The method achieves the best results on SumMe and TVSum datasets in transfer learning settings, confirming strong generalization and summarization capability.