[论文解读] Efficient Video Classification Using Fewer Frames

Shweta Bhardwaj, Mukundhan Srinivasan+1 more|arXiv (Cornell University)|Feb 27, 2019

Human Pose and Action Recognition参考文献 25被引用 4

一句话总结

该论文提出了一种基于蒸馏的方法，用于构建计算效率更高的视频分类模型，仅处理视频帧的子集，显著降低了FLOPs和推理时间。通过训练一个轻量级学生网络来模仿处理所有帧的计算密集型教师网络，该方法在YouTube-8M数据集上实现了约90%的FLOPs减少和30%的推理速度提升，同时精度损失可忽略不计。

ABSTRACT

Recently,there has been a lot of interest in building compact models for video classification which have a small memory footprint (<1 GB). While these models are compact, they typically operate by repeated application of a small weight matrix to all the frames in a video. E.g. recurrent neural network based methods compute a hidden state for every frame of the video using a recurrent weight matrix. Similarly, cluster-and-aggregate based methods such as NetVLAD, have a learnable clustering matrix which is used to assign soft-clusters to every frame in the video. Since these models look at every frame in the video, the number of floating point operations (FLOPs) is still large even though the memory footprint is small. We focus on building compute-efficient video classification models which process fewer frames and hence have less number of FLOPs. Similar to memory efficient models, we use the idea of distillation albeit in a different setting. Specifically, in our case, a compute-heavy teacher which looks at all the frames in the video is used to train a compute-efficient student which looks at only a small fraction of frames in the video. This is in contrast to a typical memory efficient Teacher-Student setting, wherein both the teacher and the student look at all the frames in the video but the student has fewer parameters. Our work thus complements the research on memory efficient video classification. We do an extensive evaluation with three types of models for video classification,viz.(i) recurrent models (ii) cluster-and-aggregate models and (iii) memory-efficient cluster-and-aggregate models and show that in each of these cases, a see-it-all teacher can be used to train a compute efficient see-very-little student. We show that the proposed student network can reduce the inference time by 30% and the number of FLOPs by approximately 90% with a negligible drop in the performance.

研究动机与目标

• 解决尽管显存使用量低，但视频分类模型计算成本高昂的问题。
• 在仍处理所有帧的紧凑型视频模型中，降低推理时间和FLOPs。
• 开发一种蒸馏框架，其中学生模型从能看到所有帧的教师模型学习，但学生仅处理稀疏帧子集。
• 在多种视频分类架构（RNN、NetVLAD和NeXtVLAD）上验证方法的有效性。
• 实现在低功耗设备上部署高效视频模型，且不损失精度。

提出的方法

• 训练一个计算量大的教师网络，处理视频中的每一帧以生成最终的视频表征。
• 训练一个轻量级学生网络，仅处理k帧（例如每j帧取一帧），以最小化FLOPs。
• 使用多种损失函数进行知识蒸馏：表征相似性损失（$L_{rep}$）、分类交叉熵损失（$L_{CE}$）和预测分布损失（$L_{pred}$）。
• 采用串行和并行两种训练范式，以优化学生从教师网络中学习。
• 通过特征级和输出级监督，使学生预测和隐藏表征与教师保持一致。
• 在YouTube-8M数据集上评估学生模型，使用mAP和GAP作为指标，与均匀采样基线和skyline模型进行比较。

Figure 1 : Architecture of Teacher-Student network for video classification

实验结果

研究问题

• RQ1 仅处理视频帧子集的学生模型能否实现与处理所有帧的教师模型相当的性能？
• RQ2 与随机或均匀采样相比，从完整帧教师模型蒸馏是否能提升稀疏帧学生模型的精度？
• RQ3 表征对齐和预测对齐的最优蒸馏损失组合（$L_{rep}$、$L_{CE}$、$L_{pred}$）是什么？
• RQ4 在多个时间步匹配中间表征是否比仅匹配最终表征能显著提升学生性能？
• RQ5 该蒸馏方法能否在不同视频分类架构（包括RNN、NetVLAD和NeXtVLAD）上实现泛化？

主要发现

• 当k=30时，学生模型相比完整帧教师模型（skyline模型）实现了约90%的FLOPs减少和30%的推理时间降低。
• 在所有评估模型中，包括NetVLAD和NeXtVLAD，学生模型在mAP和GAP指标上均优于均匀采样基线（Uniform-k）。
• 结合$L_{rep}$和$L_{pred}$损失时性能最佳，且串行训练策略结果最为一致。
• 在训练过程中匹配中间表征并未带来显著优势，表明仅匹配最终表征已足够。
• 尽管仅处理部分帧，学生模型仍保持接近skyline的性能：当k=30时，mAP仅下降0.9–2%，GAP仅下降0.5–0.9%。
• 该方法在多种架构上均有效，包括NeXtVLAD等内存高效模型，证明了其广泛适用性。

(a) Training with $L_{rep}$