[Paper Review] Temporal Convolution Based Action Proposal: Submission to ActivityNet 2017

Tianwei Lin, Xu Zhao+1 more|arXiv (Cornell University)|Jul 21, 2017

Human Pose and Action Recognition14 references63 citations

TL;DR

The paper proposes a temporal action proposal model (Prop-SSAD) based on temporal convolutions and anchor mechanisms, augmented by TAG proposals and refined boundaries, to achieve state-of-the-art results on ActivityNet 2017 for both proposal and localization tasks.

ABSTRACT

In this notebook paper, we describe our approach in the submission to the temporal action proposal (task 3) and temporal action localization (task 4) of ActivityNet Challenge hosted at CVPR 2017. Since the accuracy in action classification task is already very high (nearly 90% in ActivityNet dataset), we believe that the main bottleneck for temporal action localization is the quality of action proposals. Therefore, we mainly focus on the temporal action proposal task and propose a new proposal model based on temporal convolutional network. Our approach achieves the state-of-the-art performances on both temporal action proposal task and temporal action localization task.

Motivation & Objective

• Motivate that proposal quality is the main bottleneck for temporal action localization on untrimmed videos.
• Introduce a proposal model based on temporal convolution with anchor mechanism (Prop-SSAD).
• Augment proposals with Temporal Actionness Grouping (TAG) and refine boundaries to improve recall at high IoU.
• Demonstrate state-of-the-art performance on both temporal action proposal and localization tasks in ActivityNet 2017.
• Show that using video-level classification results can yield competitive temporal action localization results.

Proposed method

• Extract snippet-level two-stream features (appearance and motion) from untrimmed videos and resize to length 256.
• Use Prop-SSAD, a temporal anchor-based detector using multiple temporal feature maps (seven maps with lengths 1,2,4,8,16,32,64) to predict action presence and boundaries.
• Train with overlap loss for proposals and use anchor-based predictions to form proposals without initial classification.
• Implement Temporal Actionness Grouping (TAG) with an MLP to produce actionness scores and generate additional proposals.
• Refine Prop-SSAD boundaries by replacing with TAG proposals that have maximum IoU > 0.75, yielding refined proposals.
• For localization, assign video-level action categories to proposals using video-level classification results and evaluate using standard mAP across IoU thresholds.

Experimental results

Research questions

• RQ1 Can a temporal convolutional, anchor-based framework (Prop-SSAD) generate high-quality temporal action proposals without external data?
• RQ2 Does combining Prop-SSAD proposals with TAG proposals improve proposal recall, especially at higher IoU thresholds?
• RQ3 To what extent do refined boundaries influence temporal action localization performance on ActivityNet 2017?
• RQ4 Is end-to-end training feasible and beneficial for the proposed framework within ActivityNet 2017?

Key findings

• Prop-SSAD outperforms the baseline and TAG refinement improves recall, especially at higher IoU thresholds.
• Refined Prop-SSAD achieves higher AR-AN scores than Prop-SSAD (e.g., AR-AN improved from 61.52 to 64.40 in Table 1).
• Proposals refined by TAG lead to better localization results when combined with video-level classification results for action categories.
• On the validation set, their localization results show competitive or superior mAP across IoU thresholds compared to prior methods, and their testing-set average mAP is notably higher (32.26) than several baselines.
• Using first N proposals for localization indicates that localization mAP benefits from higher-quality early proposals (e.g., Ours@1–Ours@100 show progressive gains).
• The study concludes that anchor mechanisms and temporal convolution are effective for temporal action proposal tasks.