[Paper Review] ACM-Net: Action Context Modeling Network for Weakly-Supervised Temporal Action Localization

Sanqing Qu, Guang Chen+4 more|arXiv (Cornell University)|Apr 7, 2021

Human Pose and Action RecognitionComputer Science53 references44 citations

TL;DR

ACM-Net introduces a three-branch action-context attention mechanism to separate action instances, contexts, and non-action backgrounds under video-level supervision, achieving state-of-the-art weakly-supervised temporal action localization on THUMOS-14 and ActivityNet-1.3. It also attains performance comparable to some fully-supervised methods.

ABSTRACT

Weakly-supervised temporal action localization aims to localize action instances temporal boundary and identify the corresponding action category with only video-level labels. Traditional methods mainly focus on foreground and background frames separation with only a single attention branch and class activation sequence. However, we argue that apart from the distinctive foreground and background frames there are plenty of semantically ambiguous action context frames. It does not make sense to group those context frames to the same background class since they are semantically related to a specific action category. Consequently, it is challenging to suppress action context frames with only a single class activation sequence. To address this issue, in this paper, we propose an action-context modeling network termed ACM-Net, which integrates a three-branch attention module to measure the likelihood of each temporal point being action instance, context, or non-action background, simultaneously. Then based on the obtained three-branch attention values, we construct three-branch class activation sequences to represent the action instances, contexts, and non-action backgrounds, individually. To evaluate the effectiveness of our ACM-Net, we conduct extensive experiments on two benchmark datasets, THUMOS-14 and ActivityNet-1.3. The experiments show that our method can outperform current state-of-the-art methods, and even achieve comparable performance with fully-supervised methods. Code can be found at https://github.com/ispc-lab/ACM-Net

Motivation & Objective

• Motivate better suppression of semantically ambiguous action-context frames beyond simple foreground/background separation.
• Propose ACM-Net with a three-branch class-agnostic attention module to create CAS_ins, CAS_con, and CAS_bak for action instances, contexts, and backgrounds.
• Leverage MIL with three separate CAS branches to optimize video-level action classification under only video-level labels.
• Incorporate auxiliary losses (attention guide, feature separation, sparse attention) to improve snippet-level discrimination and localization.
• Demonstrate state-of-the-art or competitive performance on THUMOS-14 and ActivityNet-1.3 under weak supervision.

Proposed method

• Divide videos into non-overlapping snippets and extract RGB and Flow features as F(t).
• Embed features via a trainable convolution-based embedding to X.
• Compute initial Class Activation Sequence Phi = MLP(X).
• Apply a three-branch attention module to obtain att_ins, att_con, att_bak through a softmax Conv(X).
• Construct CAS_ins = att_ins * CAS, CAS_con = att_con * CAS, CAS_bak = att_bak * CAS.
• Use MIL by aggregating top-k scores to obtain video-level class probabilities p_ins, p_con, p_bak and compute corresponding cross-entropy losses.
• Combine losses L_cls = L_cls_ins + L_cls_con + L_cls_bak with auxiliary losses L_gui, L_feat, L_spa for guidance, feature separation, and sparsity.
• During inference, classify with p_ins and localize actions by thresholding CAS_ins and att_ins and applying NMS; refine scores with Outer-Inner-Contrastive.

Experimental results

Research questions

• RQ1 Can action-context frames be effectively separated from action instances and non-action backgrounds under weak supervision?
• RQ2 Does a three-branch attention mechanism improve discrimination between action instances, contexts, and backgrounds compared to foreground-background approaches?
• RQ3 How do auxiliary losses influence snippet-level discrimination and overall localization performance?
• RQ4 What performance gains does ACM-Net achieve on standard weakly-supervised TAL benchmarks (THUMOS-14, ActivityNet-1.3) relative to prior methods?

Key findings

• On THUMOS-14, ACM-Net achieves 68.9 mAP at 0.1 IoU and 62.7 at 0.2 IoU, with 55.0 at 0.3, 44.6 at 0.4, 34.6 at 0.5, 21.8 at 0.6, and 10.8 at 0.7 IoU, surpassing prior weakly-supervised methods.
• On THUMOS-14, ACM-Net attains 53.2 Avg[0.1-0.5] and 33.4 Avg[0.3-0.7], with an overall Avg of 42.6.
• On ActivityNet-1.3, ACM-Net achieves 40.1 mAP at IoU=0.50, 24.2 at IoU=0.75, and 6.2 at IoU=0.95, with Avg 24.6.
• ACM-Net outperforms state-of-the-art weakly-supervised TAL methods on ActivityNet-1.3 and shows competitive performance against some fully-supervised baselines at certain IoU ranges.
• Ablation analyses indicate the three-branch attention and CAS constructions are effective for suppressing context frames and improving localization accuracy.
• Qualitative visualizations demonstrate ACM-Net’s capability to distinguish action instances from ambiguous contexts and backgrounds.