[Paper Review] Temporal Action Detection with Structured Segment Networks

Yue Zhao, Yuanjun Xiong+4 more|arXiv (Cornell University)|Apr 20, 2017

Human Pose and Action Recognition48 references144 citations

TL;DR

SSN introduces structured temporal pyramid pooling to model start, course, and end stages of actions, with separate activity and completeness classifiers, plus TAG proposals, achieving state-of-the-art results on THUMOS’14 and ActivityNet.

ABSTRACT

Detecting actions in untrimmed videos is an important yet challenging task. In this paper, we present the structured segment network (SSN), a novel framework which models the temporal structure of each action instance via a structured temporal pyramid. On top of the pyramid, we further introduce a decomposed discriminative model comprising two classifiers, respectively for classifying actions and determining completeness. This allows the framework to effectively distinguish positive proposals from background or incomplete ones, thus leading to both accurate recognition and localization. These components are integrated into a unified network that can be efficiently trained in an end-to-end fashion. Additionally, a simple yet effective temporal action proposal scheme, dubbed temporal actionness grouping (TAG) is devised to generate high quality action proposals. On two challenging benchmarks, THUMOS14 and ActivityNet, our method remarkably outperforms previous state-of-the-art methods, demonstrating superior accuracy and strong adaptivity in handling actions with various temporal structures.

Motivation & Objective

• Model the temporal structure of action instances using a three-stage (starting, course, ending) representation.
• Develop a structured temporal pyramid pooling (STPP) to form rich global proposals.
• Jointly learn action classification and completeness assessment to filter incomplete or background proposals.
• Enable end-to-end training with sparse snippet sampling for efficiency on long videos.
• Propose a robust temporal action proposal method (Temporal Actionness Grouping, TAG) to improve recall and precision.

Proposed method

• Split augmented proposals into starting, course, and ending stages and apply structured temporal pyramid pooling to obtain stage-wise representations.
• Use a two-classifier system: an activity classifier for K+1 classes (including background) and a set of K completeness classifiers to assess proposal completeness.
• Train with a multi-task loss combining classification and completeness, plus a location regression for temporal bounds.
• Adopt sparse snippet sampling (L=9 segments) to approximate STPP efficiently during training.
• During inference, reorder computation to reuse shared features and accelerate classification/regression for many proposals.
• Generate proposals with Temporal Actionness Grouping (TAG) by flooding a 1D actionness signal and merging basins to form high-quality proposals.

Experimental results

Research questions

• RQ1 Can explicit temporal structure modeling (start/course/end) distinguish complete action instances from incomplete or background proposals?
• RQ2 Does structured temporal pyramid pooling improve representations for long, variably structured actions compared to flat pooling?
• RQ3 How do separate activity and completeness classifiers contribute to detection accuracy and precision at various IoU thresholds?
• RQ4 What is the impact of end-to-end training with sparse sampling on performance and efficiency?
• RQ5 Does TAG provide higher recall and better proposal quality than traditional sliding windows or other proposals in temporal action detection?

Key findings

• SSN achieves state-of-the-art results on THUMOS’14 and ActivityNet v1.3/1.2, outperforming prior methods across multiple metrics.
• Structured temporal pyramid pooling with augmented proposals (start, course, end) improves detection over setups without stage structure.
• Decomposed activity and completeness classifiers outperform a single unified negative-class approach.
• Location regression and multi-task training consistently improve detection accuracy.
• End-to-end training with sparse sampling yields competitive performance with reduced storage and computation.
• TAG proposals provide higher recall and better IoU quality, enhancing overall detection performance.