[Paper Review] Temporal Context Network for Activity Localization in Videos

Xiyang Dai, Bharat Singh+3 more|arXiv (Cornell University)|Aug 8, 2017

Human Pose and Action Recognition33 references66 citations

TL;DR

Introduces Temporal Context Network (TCN) that ranks temporally proposed segments using context from a larger scale, achieving state-of-the-art results on ActivityNet and THUMOS14 by explicit multi-scale context sampling and temporal convolution.

ABSTRACT

We present a Temporal Context Network (TCN) for precise temporal localization of human activities. Similar to the Faster-RCNN architecture, proposals are placed at equal intervals in a video which span multiple temporal scales. We propose a novel representation for ranking these proposals. Since pooling features only inside a segment is not sufficient to predict activity boundaries, we construct a representation which explicitly captures context around a proposal for ranking it. For each temporal segment inside a proposal, features are uniformly sampled at a pair of scales and are input to a temporal convolutional neural network for classification. After ranking proposals, non-maximum suppression is applied and classification is performed to obtain final detections. TCN outperforms state-of-the-art methods on the ActivityNet dataset and the THUMOS14 dataset.

Motivation & Objective

• Motivate precise temporal localization of human activities in untrimmed videos.
• Develop a proposal ranking mechanism that incorporates temporal context beyond the proposal boundaries.
• Leverage a two-stream feature representation with context sampling across scales.
• Integrate proposal ranking with a classification stage to yield final detections.
• Demonstrate improvements over prior methods on ActivityNet and THUMOS14.

Proposed method

• Place proposals at equal intervals and across multiple temporal scales to cover potential activity regions.
• Construct a context-aware feature representation by sampling features inside the proposal and from the next larger scale, then concatenate them.
• Apply temporal convolution over paired scale features to predict proposal foreground/background.
• Use bilinear pooling over proposal features for robust action classification, followed by a signed square root and L2 normalization.
• Train with cross-entropy losses for both the proposal ranker and the final classifier, using balanced mini-batches.
• Evaluate with non-maximum suppression to finalize detections.

Experimental results

Research questions

• RQ1 Does incorporating temporal context from larger scales improve the ranking of temporal activity proposals?
• RQ2 How does the scale of context windows affect localization performance and mAP across IoU thresholds?
• RQ3 Can a proposal-based detection system with context-based ranking outperform LSTM-based or global-feature approaches on untrimmed videos?
• RQ4 What is the impact of the number of proposals on detection performance in ActivityNet and THUMOS14?

Key findings

• Temporal context significantly improves proposal ranking, especially at higher IoU thresholds.
• Context-based proposals yield higher mAP than context-less counterparts across multiple thresholds (ActivityNet validation): mAP@.5=36.17, mAP@.75=21.12, mAP@.95=3.89 when using context vs 15.91, 3.11, 0.13 without context.
• Using a context scale around 2 provides best performance, with larger or smaller scales reducing effectiveness.
• Top-performing results on ActivityNet evaluation server: mAP@.5=37.49, mAP@.75=23.47, mAP@.95=4.47.
• On THUMOS14, the method achieves higher average recalls and improves detection performance when paired with CDC, outperforming several baselines.
• The proposed ranker with context outperforms prior proposal methods (e.g., DAPs, SCNN) across recall and detection metrics.