[Paper Review] Temporal Sequence Distillation: Towards Few-Frame Action Recognition in Videos

Zhaoyang Zhang, Zhanghui Kuang+3 more|arXiv (Cornell University)|Aug 15, 2018

Human Pose and Action Recognition22 references4 citations

TL;DR

This paper proposes Temporal Sequence Distillation (TSD), a method that distills long video sequences into compact, representative short sequences using end-to-end 3D CNN training. TSD enables high-accuracy few-frame action recognition by learning discriminative spatiotemporal representations, reducing frame transmission by up to 50% and cloud FLOPs by 50% while maintaining state-of-the-art accuracy on Kinetics and UCF101 datasets.

ABSTRACT

Video Analytics Software as a Service (VA SaaS) has been rapidly growing in recent years. VA SaaS is typically accessed by users using a lightweight client. Because the transmission bandwidth between the client and cloud is usually limited and expensive, it brings great benefits to design cloud video analysis algorithms with a limited data transmission requirement. Although considerable research has been devoted to video analysis, to our best knowledge, little of them has paid attention to the transmission bandwidth limitation in SaaS. As the first attempt in this direction, this work introduces a problem of few-frame action recognition, which aims at maintaining high recognition accuracy, when accessing only a few frames during both training and test. Unlike previous work that processed dense frames, we present Temporal Sequence Distillation (TSD), which distills a long video sequence into a very short one for transmission. By end-to-end training with 3D CNNs for video action recognition, TSD learns a compact and discriminative temporal and spatial representation of video frames. On Kinetics dataset, TSD+I3D typically requires only 50\% of the number of frames compared to I3D, a state-of-the-art video action recognition algorithm, to achieve almost the same accuracies. The proposed TSD has three appealing advantages. Firstly, TSD has a lightweight architecture and can be deployed in the client, eg. mobile devices, to produce compressed representative frames to save transmission bandwidth. Secondly, TSD significantly reduces the computations to run video action recognition with compressed frames on the cloud, while maintaining high recognition accuracies. Thirdly, TSD can be plugged in as a preprocessing module of any existing 3D CNNs. Extensive experiments show the effectiveness and characteristics of TSD.

Motivation & Objective

• To address the bandwidth and computational cost in Video Analytics as a Service (VA SaaS) by enabling high-accuracy action recognition with minimal frame transmission.
• To develop a method that learns compact, representative video sequences from long input videos, preserving discriminative temporal and spatial features.
• To enable deployment of the distillation model on lightweight clients (e.g., mobile devices) to reduce data transmission and server-side computation.
• To maintain high recognition accuracy even when only a small number of frames are transmitted to the cloud.
• To provide a plug-in preprocessing module compatible with existing 3D CNNs like I3D for efficient integration in real-world VA SaaS pipelines.

Proposed method

• TSD uses an end-to-end training pipeline with a 3D CNN backbone (e.g., I3D) to learn to compress long video sequences into short, representative sequences.
• The method includes a coarse feature extractor and a temporal sequence distillation block that jointly learn to identify key frames relevant to action classes.
• TSD is trained via knowledge distillation, where the model learns to mimic the output of a full-clip 3D CNN using only a subset of frames.
• The distillation process preserves temporal dynamics and spatial context by focusing on action-relevant frames, reducing redundancy while maintaining discriminative power.
• The framework is modular and can be plugged in as a preprocessing step before any 3D CNN, enabling compatibility with existing models.
• During inference, the client runs TSD to generate a compact frame sequence, which is transmitted to the cloud for final classification.

Experimental results

Research questions

• RQ1 Can a compact, representative video sequence be learned end-to-end to maintain high action recognition accuracy with minimal frame transmission?
• RQ2 How does TSD compare to uniform sampling and other frame selection strategies in terms of accuracy and efficiency on standard benchmarks?
• RQ3 To what extent can TSD reduce data transmission and computational cost in VA SaaS scenarios without sacrificing recognition performance?
• RQ4 Can TSD be effectively deployed on client-side devices and integrated with existing 3D CNNs like I3D?
• RQ5 What is the trade-off between frame reduction, computational savings, and accuracy across different datasets and model configurations?

Key findings

• On the Kinetics dataset, I3D+TSD achieves 72.4% accuracy using only 120 transmitted frames—50% fewer than I3D’s 250 frames—while maintaining comparable performance.
• When reducing frame transmission to 60 frames, I3D+TSD achieves 70.7% accuracy, outperforming the baseline I3D (68.4%) at 40 frames and significantly reducing cloud FLOPs by 51%.
• On UCF101, I3D+TSD improves accuracy by 7.7% over I3D when only 5 frames are sampled, demonstrating strong performance in few-frame regimes.
• I3D+TSD reduces cloud-side FLOPs by up to 50% and inference time by up to 48% compared to I3D, even with a 33% increase in model parameters.
• The method maintains consistent performance gains across different clip numbers and frame sampling rates, showing robustness and scalability.
• TSD’s client-side deployment reduces transmission bandwidth and cloud computation, making it highly suitable for real-time VA SaaS applications.