[论文解读] Equivalent Classification Mapping for Weakly Supervised Temporal Action Localization

Tao Zhao, Junwei Han+2 more|arXiv (Cornell University)|Aug 18, 2020

Human Pose and Action Recognition被引用 7

一句话总结

该论文提出等价分类映射（ECM），一种新颖的弱监督动作定位框架，通过训练共享同一分类器的两个并行分支，统一了预分类与后分类流水线。通过强制帧级得分与特征聚合权重之间的一致性，并引入权重转移模块与等价训练策略，ECM在THUMOS14数据集上实现了54.3%的帧准确率，显著优于基线方法与模型集成方法。

ABSTRACT

Weakly supervised temporal action localization is a newly emerging yet widely studied topic in recent years. The existing methods can be categorized into two localization-by-classification pipelines, i.e., the pre-classification pipeline and the post-classification pipeline. The pre-classification pipeline first performs classification on each video snippet and then aggregate the snippet-level classification scores to obtain the video-level classification score. In contrast, the post-classification pipeline aggregates the snippet-level features first and then predicts the video-level classification score based on the aggregated feature. Although the classifiers in these two pipelines are used in different ways, the role they play is exactly the same---to classify the given features to identify the corresponding action categories. To this end, an ideal classifier can make both pipelines work. This inspires us to simultaneously learn these two pipelines in a unified framework to obtain an effective classifier. Specifically, in the proposed learning framework, we implement two parallel network streams to model the two localization-by-classification pipelines simultaneously and make the two network streams share the same classifier. This achieves the novel Equivalent Classification Mapping (ECM) mechanism. Moreover, we discover that an ideal classifier may possess two characteristics: 1) The frame-level classification scores obtained from the pre-classification stream and the feature aggregation weights in the post-classification stream should be consistent; 2) The classification results of these two streams should be identical. Based on these two characteristics, we further introduce a weight-transition module and an equivalent training strategy into the proposed learning framework, which assists to thoroughly mine the equivalence mechanism.

研究动机与目标

• 解决缺乏片段级标注时的弱监督时序动作定位挑战。
• 将两种主流流水线——预分类与后分类——统一为一个统一且连贯的学习框架。
• 学习一个共享分类器，以保持帧级预测与聚合视频级特征之间的等价性。
• 通过在不同流水线间强制分类得分与注意力权重之间的一致性，提升定位准确率。
• 建立一个简单但有效的弱监督动作定位基线，具备跨基准的泛化能力。

提出的方法

• 提出一种双流网络架构，包含共享同一分类器的并行预分类与后分类流水线。
• 引入一种等价训练策略，通过同步两个分支的梯度以强制分类器一致性。
• 设计一个权重转移模块，以对齐后分类流水线中帧级分类得分与特征聚合权重。
• 使用视频级分类损失 $\mathcal{L}_{\text{cls}}$ 同时监督两个分支，确保端到端学习。
• 采用统一分类器，从点级特征与聚合视频级特征中预测动作类别。
• 应用t-SNE可视化以确认聚合特征位于与动作特征相同的流形上，支持等价性假设。

实验结果

研究问题

• RQ1 共享分类器能否有效统一弱监督时序动作定位中的预分类与后分类流水线？
• RQ2 强制帧级得分与聚合权重之间的一致性是否能提升定位准确率？
• RQ3 所提出的等价机制能否超越模型集成与单一流水线基线？
• RQ4 权重转移模块在对齐两个流水线中起到了何种作用？
• RQ5 所提出的框架在多个基准上是否具备鲁棒性与泛化能力？

主要发现

• ECM在THUMOS14数据集上实现了54.3%的帧准确率，显著优于预分类流水线（42.7%）与后分类流水线（39.0%）。
• ECM与模型集成（29.1 vs. 21.4 mAP）之间的性能差距表明，等价机制优于简单集成。
• 与直接实现等价性的方法相比，权重转移模块与等价训练策略联合提升了8.7 mAP的性能。
• 定性可视化显示，ECM生成了更清晰、更具判别力的激活序列，能准确定位动作实例并抑制背景响应。
• t-SNE可视化证实，聚合的视频级特征位于与动作特征相同的特征流形上，验证了ECM的理论基础。
• ECM在无需架构复杂性或额外组件的情况下达到最先进性能，为未来研究建立了强大而简洁的基线。