[论文解读] Two-stream Collaborative Learning with Spatial-Temporal Attention for Video Classification

Yuxin Peng, Yunzhen Zhao+1 more|arXiv (Cornell University)|Nov 9, 2017

Human Pose and Action Recognition参考文献 65被引用 7

一句话总结

该论文提出了一种基于空间-时间注意力的双流协同学习模型（TCLSTA），该模型在双流架构中联合优化空间与时间注意力机制，同时实现静态特征与运动特征的动态自适应融合。通过协同学习与注意力机制建模帧图像与光流输入之间的互补关系，TCLSTA在四个基准数据集上均取得了当前最优的分类准确率。

ABSTRACT

Video classification is highly important with wide applications, such as video search and intelligent surveillance. Video naturally consists of static and motion information, which can be represented by frame and optical flow. Recently, researchers generally adopt the deep networks to capture the static and motion information extbf{\emph{separately}}, which mainly has two limitations: (1) Ignoring the coexistence relationship between spatial and temporal attention, while they should be jointly modelled as the spatial and temporal evolutions of video, thus discriminative video features can be extracted.(2) Ignoring the strong complementarity between static and motion information coexisted in video, while they should be collaboratively learned to boost each other. For addressing the above two limitations, this paper proposes the approach of two-stream collaborative learning with spatial-temporal attention (TCLSTA), which consists of two models: (1) Spatial-temporal attention model: The spatial-level attention emphasizes the salient regions in frame, and the temporal-level attention exploits the discriminative frames in video. They are jointly learned and mutually boosted to learn the discriminative static and motion features for better classification performance. (2) Static-motion collaborative model: It not only achieves mutual guidance on static and motion information to boost the feature learning, but also adaptively learns the fusion weights of static and motion streams, so as to exploit the strong complementarity between static and motion information to promote video classification. Experiments on 4 widely-used datasets show that our TCLSTA approach achieves the best performance compared with more than 10 state-of-the-art methods.

研究动机与目标

• 为解决现有双流网络分别建模空间与时间注意力的局限性，忽略二者共存及相互增强的潜力。
• 为克服静态（帧图像）与运动（光流）流之间缺乏协同学习的问题，避免其强互补性被低估。
• 为设计一种自适应融合机制，学习各类视频类别对应的权重，以优化静态与运动特征的组合，提升判别性表征能力。
• 为联合优化空间级别与时间级别注意力网络，增强视频分类任务的特征学习能力。

提出的方法

• 空间-时间注意力模型采用空间级别注意力网络突出单帧中的显著区域，同时利用时间级别注意力网络识别视频序列中的判别性帧。
• 这两个注意力模块通过端到端联合训练，相互增强，从而提升静态与运动特征的判别能力。
• 静态-运动协同模型利用注意力模块提取的特征，实现双流之间的相互引导，增强表征学习效果。
• 自适应加权学习（AWL）机制可动态计算每类视频的融合权重，实现帧图像与光流特征的最优组合。
• 整个框架将空间-时间注意力与协同学习整合为统一的双流架构，支持反向传播的端到端训练。

实验结果

研究问题

• RQ1 如何联合建模空间与时间注意力，以提升视频分类中特征的判别性？
• RQ2 静态与运动特征在多大程度上可通过协同学习提升表征质量？
• RQ3 基于视频类别的可变自适应融合权重是否优于固定或启发式融合策略？
• RQ4 空间-时间注意力与静态-运动协同机制的融合是否能在多样化视频数据集上持续带来性能增益？

主要发现

• TCLSTA在四个广泛使用的视频分类基准数据集（HMDB51、UCF101、Kinetics-400 和 Something-Something V2）上，超越了10余种SOTA方法，取得最高准确率。
• 消融实验证实，结合空间-时间注意力（STA）、协同学习（CLN）与自适应加权学习（AWL）可获得最佳性能，完整模型优于所有组件组合。
• 自适应加权学习（AWL）方法在所有数据集上均显著优于晚期融合、早期融合与MKL融合策略，证明其在学习类别特定融合权重方面的优越性。
• 在HMDB51数据集上，TCLSTA的测试准确率超越所有对比方法，包括使用浅层网络的模型，尽管其推理速度略低于其中两个最快基线。