[Paper Review] UC Merced Submission to the ActivityNet Challenge 2016

Yi Zhu, Shawn Newsam+1 more|arXiv (Cornell University)|Apr 11, 2017

Human Pose and Action Recognition15 references7 citations

TL;DR

This paper presents a late-fusion multi-stream framework for untrimmed video action recognition in the ActivityNet Challenge 2016, combining hand-crafted MBH features with deep network activations from VGG16, GoogLeNet, C3D, and ResNet-101. The system achieves a top-1 accuracy of 75.14% on the validation set and 78.44% on the test set by fusing models via weighted averaging and iterative re-ranking, demonstrating that complementary low-level motion features and deep learning representations significantly improve performance.

ABSTRACT

This notebook paper describes our system for the untrimmed classification task in the ActivityNet challenge 2016. We investigate multiple state-of-the-art approaches for action recognition in long, untrimmed videos. We exploit hand-crafted motion boundary histogram features as well feature activations from deep networks such as VGG16, GoogLeNet, and C3D. These features are separately fed to linear, one-versus-rest support vector machine classifiers to produce confidence scores for each action class. These predictions are then fused along with the softmax scores of the recent ultra-deep ResNet-101 using weighted averaging.

Motivation & Objective

• To develop a robust action recognition system for long, untrimmed videos in the ActivityNet Challenge 2016.
• To investigate the effectiveness of combining hand-crafted motion boundary histogram (MBH) features with deep learning-based features from multiple architectures.
• To improve recognition accuracy through late fusion of diverse feature streams and iterative re-ranking.
• To evaluate the impact of using action proposals versus uniform frame sampling for feature extraction.
• To demonstrate the superiority of ultra-deep residual networks (ResNet-101) and the complementarity of MBH with deep features.

Proposed method

• The system employs a multi-stream framework fusing five components: Fisher vector-encoded MBH features, C3D fc7 features, GoogLeNet pool5 features, VGG16 pool5 features, and ResNet-101 softmax scores.
• MBH features are extracted from pre-computed motion boundaries and encoded using Fisher vector coding, followed by SVM classification with C=100.
• C3D features are extracted from 16-frame clips with 50% overlap, reduced via PCA to 500 dimensions, and averaged across clips before SVM classification with C=1.
• GoogLeNet and VGG16 features are extracted frame-by-frame, mean-pooled across the video, and normalized; VGG16 features are further encoded using LCD and VLAD with PCA reduction.
• ResNet-101 features are fine-tuned on the dataset, and its softmax scores are used directly in fusion instead of extracted features, which performed worse.
• Late fusion combines model outputs via weighted averaging, with more accurate models weighted twice as much, followed by Multi-class Iterative Re-ranking (MIR) to improve predictions on hard examples.

Experimental results

Research questions

• RQ1 How do hand-crafted motion boundary histogram (MBH) features compare to deep learning-based features in untrimmed video action recognition?
• RQ2 Can late fusion of multiple deep network streams and MBH features improve recognition accuracy beyond individual models?
• RQ3 Does using action proposals for frame sampling outperform uniform sampling in feature extraction for video classification?
• RQ4 How effective is the ResNet-101 architecture for action recognition in long, untrimmed videos compared to shallower networks?
• RQ5 To what extent are MBH features complementary to deep network features in capturing spatio-temporal information?

Key findings

• The ResNet-101 model achieved the highest single-model top-1 accuracy of 71.81% on the validation set, significantly outperforming other models.
• The combination of all five modules via late fusion achieved a top-1 accuracy of 75.14% on the validation set, demonstrating the effectiveness of multi-stream fusion.
• Incorporating MBH features improved performance even though they performed worst individually, indicating strong complementarity with deep features.
• Fusing only deep network features (e.g., VGG16, GoogLeNet, C3D) yielded lower performance than fusing them with MBH, highlighting MBH’s unique contribution.
• Using action proposals for frame sampling instead of uniform sampling degraded performance, suggesting that uniform sampling better captures representative video content.
• The final submission (Run 5) achieved 83.1% mAP and 78.44% top-1 accuracy on the test set, ranking among the top entries in the ActivityNet Challenge 2016.