[Paper Review] Untrimmed Video Classification for Activity Detection: submission to ActivityNet Challenge

Gurkirt Singh, Fabio Cuzzolin|arXiv (Cornell University)|Jul 7, 2016

Human Pose and Action RecognitionComputer Science7 references100 citations

TL;DR

The paper proposes a two-stage approach that uses untrimmed video-level classification to guide frame-level proposal generation via dynamic programming for temporal activity detection in untrimmed videos, achieving runner-up in ActivityNet 2016.

ABSTRACT

Current state-of-the-art human activity recognition is focused on the classification of temporally trimmed videos in which only one action occurs per frame. We propose a simple, yet effective, method for the temporal detection of activities in temporally untrimmed videos with the help of untrimmed classification. Firstly, our model predicts the top k labels for each untrimmed video by analysing global video-level features. Secondly, frame-level binary classification is combined with dynamic programming to generate the temporally trimmed activity proposals. Finally, each proposal is assigned a label based on the global label, and scored with the score of the temporal activity proposal and the global score. Ultimately, we show that untrimmed video classification models can be used as stepping stone for temporal detection.

Motivation & Objective

• Motivate temporal activity detection in temporally untrimmed videos beyond single-action frames.
• Propose a simple fusion-based untrimmed classification pipeline to generate activity proposals.
• Demonstrate that untrimmed video classification can serve as a stepping stone for temporal detection.
• Evaluate on ActivityNet to show competitive performance and discuss extensions to online detection.

Proposed method

• Extract video-level features (Imagenet Shuffle, MBH global features, and mean-pooled C3D frame-level scores).
• Train one-vs-rest linear SVMs per class on each feature type to obtain scores S^i, S^m, S^3.
• Fuse video-level scores with a linear SVM meta-classifier over stacked scores to obtain untrimmed classification scores S^s.
• Train per-class binary random forest classifiers on frame-level C3D features to obtain frame-level positive scores s^r_t.
• Formulate activity proposal generation as a DP optimization that selects a piecewise-constant labeling to maximize the sum of frame scores minus a boundary penalty.
• Assign the top activity proposals a class label and combine with the global video-class score to produce detections.

Experimental results

Research questions

• RQ1 Can untrimmed video-level classification be leveraged to detect temporal activity bounds in untrimmed videos?
• RQ2 How effective is a DP-based framework to generate accurate activity proposals from frame-level binary decisions?
• RQ3 Does combining local (frame-level) and global (video-level) scores improve activity detection performance on ActivityNet?
• RQ4 What is the impact of normalizing SVM scores by the top-k scores on untrimmed classification performance?

Key findings

• Untrimmed classification with the proposed fusion achieves high TOP-1 and TOP-3 on validation and strong results on testing for untrimmed classification.
• On the ActivityNet challenge, the proposed approach outperforms the cited baseline in validation (TOP-1 76.89%, TOP-3 89.25%, mAP 81.99%) and achieves testing TOP-1 77.08%, TOP-3 89.38%, mAP 82.49%.
• Activity proposals generated by a DP-based optimization align well with ground truth, enabling effective temporal localization.
• Top-2 activity proposals per video are labeled with the top untrimmed classification class and scored by the product of the global and proposal scores (S_{c}^{s} * S_{a}).
• Dynamic programming provides an efficient solution for frame-level binary decisions to produce contiguous activity proposals.
• Results include TIoU-based evaluation showing notable gains over baselines (validation: 0.1–0.5 TIoU thresholds).
• The method demonstrates potential for online detection and simultaneous detection/classification extensions.