[Paper Review] Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition

Haisheng Su, Jing Su+6 more|arXiv (Cornell University)|Sep 15, 2020

Human Pose and Action Recognition32 references9 citations

TL;DR

This paper proposes Feature and Parameter Collaborative Distillation (FPCD), a novel knowledge distillation framework for video action recognition that leverages temporal frequency spectra of features and parameter distributions in the frequency domain to train lightweight student networks. By jointly distilling motion dynamics and appearance modeling knowledge from a teacher network, FPCD achieves state-of-the-art performance on Kinetics, Something-Something, and Jester with significantly reduced model size and computation.

ABSTRACT

Recent years have witnessed the significant progress of action recognition task with deep networks. However, most of current video networks require large memory and computational resources, which hinders their applications in practice. Existing knowledge distillation methods are limited to the image-level spatial domain, ignoring the temporal and frequency information which provide structural knowledge and are important for video analysis. This paper explores how to train small and efficient networks for action recognition. Specifically, we propose two distillation strategies in the frequency domain, namely the feature spectrum and parameter distribution distillations respectively. Our insight is that appealing performance of action recognition requires extit{explicitly} modeling the temporal frequency spectrum of video features. Therefore, we introduce a spectrum loss that enforces the student network to mimic the temporal frequency spectrum from the teacher network, instead of extit{implicitly} distilling features as many previous works. Second, the parameter frequency distribution is further adopted to guide the student network to learn the appearance modeling process from the teacher. Besides, a collaborative learning strategy is presented to optimize the training process from a probabilistic view. Extensive experiments are conducted on several action recognition benchmarks, such as Kinetics, Something-Something, and Jester, which consistently verify effectiveness of our approach, and demonstrate that our method can achieve higher performance than state-of-the-art methods with the same backbone.

Motivation & Objective

• Address the challenge of deploying large, computationally expensive video action recognition models in real-world applications with limited resources.
• Overcome the limitations of existing knowledge distillation methods that focus only on spatial features and ignore temporal and frequency-domain structural knowledge in videos.
• Improve performance of lightweight student networks by explicitly modeling temporal dynamics and appearance representations through frequency-domain knowledge.
• Develop a collaborative learning strategy to dynamically eliminate redundant knowledge during distillation for more efficient training.
• Demonstrate generalizability across diverse video architectures and benchmarks, including Kinetics, Something-Something, and Jester.

Proposed method

• Introduce a spectrum loss that encourages the student network to mimic the temporal frequency spectrum of video features from the teacher network, capturing both motion and scene-level representations.
• Propose parameter distribution distillation to transfer structural knowledge from the teacher’s convolutional layer weights by matching their frequency-domain distributions.
• Formulate a collaborative learning strategy that optimizes the distillation process from a probabilistic perspective, adaptively suppressing redundant knowledge from the teacher.
• Combine the spectrum loss, parameter distillation loss, and standard cross-entropy classification loss into a unified training objective.
• Apply the framework to various video backbone architectures (e.g., STM, I3D, TSN) and datasets to evaluate generalization and efficiency.
• Use Fourier transforms to analyze and compare feature and parameter frequency spectra, validating the importance of low- and high-frequency components in video understanding.

Experimental results

Research questions

• RQ1 Can distilling temporal frequency spectra of video features improve action recognition performance in lightweight student networks?
• RQ2 Does modeling the frequency distribution of network parameters enhance the student’s ability to learn appearance modeling from the teacher?
• RQ3 How does collaborative learning improve knowledge distillation efficiency by eliminating redundant knowledge?
• RQ4 To what extent does the proposed method generalize across different video architectures and datasets?
• RQ5 Which frequency components—low or high—contribute more to performance on different types of video datasets (e.g., scene-related vs. action-related)?

Key findings

• On Kinetics-400, FPCD with a student STM-ResNet18 achieves 71.1% top-1 accuracy, surpassing the baseline Simple KD and CCKD methods, and closing the gap with a deeper teacher ResNet-50.
• On Something-Something V1, FPCD achieves 39.1% top-1 accuracy with a channel-halved I3D backbone, outperforming Simple KD (37.4%) and CCKD (38.0%) by 1.7% and 1.1% respectively.
• The low-frequency feature spectrum distillation contributes 1.3% more performance gain than high-frequency distillation on Kinetics-400, confirming its importance for scene appearance modeling.
• Parameter distribution distillation yields greater improvements than feature spectrum distillation on scene-related datasets like Kinetics-400, indicating its effectiveness in capturing spatial appearance cues.
• The collaborative learning strategy enables dynamic elimination of redundant knowledge, improving distillation efficiency and performance consistency across different frameworks.
• FPCD generalizes well across architectures: on TSN-ResNet18, it outperforms the teacher ResNet-50 on Jester and Something-Something V2, and maintains consistent gains on lightweight Inception V1 (channel-halved) across all datasets.