[Paper Review] Revisiting the Effectiveness of Off-the-shelf Temporal Modeling Approaches for Large-scale Video Classification

Yunlong Bian, Chuang Gan+8 more|arXiv (Cornell University)|Aug 12, 2017

Human Pose and Action Recognition18 references53 citations

TL;DR

The paper evaluates off-the-shelf temporal modeling methods for large-scale video classification using multi-modal features, proposing four models that achieve state-of-the-art results on Kinetics, especially when ensembled.

ABSTRACT

This paper describes our solution for the video recognition task of ActivityNet Kinetics challenge that ranked the 1st place. Most of existing state-of-the-art video recognition approaches are in favor of an end-to-end pipeline. One exception is the framework of DevNet. The merit of DevNet is that they first use the video data to learn a network (i.e. fine-tuning or training from scratch). Instead of directly using the end-to-end classification scores (e.g. softmax scores), they extract the features from the learned network and then fed them into the off-the-shelf machine learning models to conduct video classification. However, the effectiveness of this line work has long-term been ignored and underestimated. In this submission, we extensively use this strategy. Particularly, we investigate four temporal modeling approaches using the learned features: Multi-group Shifting Attention Network, Temporal Xception Network, Multi-stream sequence Model and Fast-Forward Sequence Model. Experiment results on the challenging Kinetics dataset demonstrate that our proposed temporal modeling approaches can significantly improve existing approaches in the large-scale video recognition tasks. Most remarkably, our best single Multi-group Shifting Attention Network can achieve 77.7% in term of top-1 accuracy and 93.2% in term of top-5 accuracy on the validation set.

Motivation & Objective

• Motivate improved large-scale video understanding using learned features followed by off-the-shelf temporal models.
• Evaluate multiple temporal modeling approaches on Kinetics using RGB, Flow, and Audio features.
• Propose four novel temporal modeling approaches and assess their complementary benefits.

Proposed method

• Extract multi-modal features (RGB, Flow, Audio) using Inception-ResNet-v2 for RGB/Flow and a VGG16-based audio model within a Temporal Segment Network framework.
• Propose four off-the-shelf temporal modeling approaches: Multi-group Shifting Attention Network, Temporal Xception Network, Multi-stream Sequence Model, and Fast-Forward Sequence Model.
• Use depthwise separable convolutions and attention-based shifting operations for temporal modeling.
• Fuse modality-specific attention/group outputs and feed to a classifier; compare to traditional temporal pooling and LSTM baselines.
• Evaluate on Kinetics with a fixed-length/segment-based testing protocol and report Top-1/Top-5 accuracy.
• Show ensemble gains from combining individual models.

Experimental results

Research questions

• RQ1 How effective are off-the-shelf temporal modeling approaches when applied to learned multi-modal video features for large-scale action recognition?
• RQ2 Can the proposed temporal models match or surpass traditional sequence models like LSTMs on Kinetics?
• RQ3 Do different temporal modeling approaches complement each other to yield better ensemble performance?
• RQ4 What is the contribution of multi-modal features (RGB, Flow, Audio) versus naive score fusion for performance gains?

Key findings

• Temporal modeling with multi-modal features outperforms naive score fusion of modality-specific classifiers.
• The proposed Shifting Attention Network and Temporal Xception Network achieve comparable or better results than traditional sequence models such as LSTMs.
• The four temporal models are complementary, and their ensemble yields the best performance.
• On Kinetics validation, the best single model (Shifting Attention Network) achieves 77.7% top-1 and 93.2% top-5 accuracy; ensemble reaches 81.5% top-1 and 95.6% top-5 accuracy.