[Paper Review] Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs

Zheng Shou, Dongang Wang+1 more|arXiv (Cornell University)|Jan 9, 2016

Human Pose and Action Recognition175 citations

TL;DR

The paper introduces a Segment-CNN framework with a proposal, classification, and localization network using 3D ConvNets to localize actions in untrimmed videos, achieving notable mAP gains on MEXaction2 and THUMOS 2014.

ABSTRACT

We address temporal action localization in untrimmed long videos. This is important because videos in real applications are usually unconstrained and contain multiple action instances plus video content of background scenes or other activities. To address this challenging issue, we exploit the effectiveness of deep networks in temporal action localization via three segment-based 3D ConvNets: (1) a proposal network identifies candidate segments in a long video that may contain actions; (2) a classification network learns one-vs-all action classification model to serve as initialization for the localization network; and (3) a localization network fine-tunes on the learned classification network to localize each action instance. We propose a novel loss function for the localization network to explicitly consider temporal overlap and therefore achieve high temporal localization accuracy. Only the proposal network and the localization network are used during prediction. On two large-scale benchmarks, our approach achieves significantly superior performances compared with other state-of-the-art systems: mAP increases from 1.7% to 7.4% on MEXaction2 and increases from 15.0% to 19.0% on THUMOS 2014, when the overlap threshold for evaluation is set to 0.5.

Motivation & Objective

• Address temporal action localization in untrimmed long videos containing multiple action instances and background content.
• Leverage multi-stage 3D ConvNets to propose, classify, and precisely localize actions in time.
• Improve temporal localization precision by a novel loss that accounts for IoU with ground truth.
• Show that a classification-initialized localization network yields better boundaries than classification alone.
• Demonstrate efficiency and accuracy improvements over state-of-the-art on large-scale benchmarks.

Proposed method

• Use multi-scale segment generation by sliding windows to create candidate segments from untrimmed videos.
• Employ a proposal network (background vs. action) to filter unlikely segments.
• Train a classification network for K action categories to initialize localization.
• Introduce a localization network with a loss that explicitly encourages higher scores for segments with greater IoU to ground truth.
• Predict with proposal and localization networks; apply NMS with adjusted thresholds for final detections.

Experimental results

Research questions

• RQ1 Can a multi-stage Segment-CNN framework improve temporal localization accuracy in untrimmed videos over prior approaches?
• RQ2 Does initializing the localization network with a classification network and using an overlap-aware loss improve IoU-aligned action boundaries?
• RQ3 How do multi-scale segment proposals, background filtering, and NMS affect final localization performance?
• RQ4 What is the impact of the localization loss parameter and the α exponent on training stability and accuracy?

Key findings

• Segment-CNN substantially improves mAP on MEXaction2 from 1.7% to 7.4%.
• On THUMOS 2014, Segment-CNN improves mAP from 15.0% to 19.0% at IoU threshold 0.5.
• The proposal network filters background segments, improving precision and efficiency.
• The localization network with the overlap-aware loss better prioritizes segments with higher ground-truth overlap.
• Classification pretraining provides a beneficial initialization for localization, yielding better results than w/o classification.
• The approach runs around 1 second per batch with modest GPU memory requirements and does not require caching high-dimensional features.