[Paper Review] Temporal Segment Networks: Towards Good Practices for Deep Action Recognition

Limin Wang, Yuanjun Xiong+5 more|arXiv (Cornell University)|Aug 2, 2016

Human Pose and Action RecognitionComputer Science32 references289 citations

TL;DR

Introduces Temporal Segment Networks (TSN) to model long-range temporal structure in videos using sparse sampling and video-level supervision, enabling deep ConvNets to achieve state-of-the-art action recognition on HMDB51 and UCF101 with practical training strategies.

ABSTRACT

Deep convolutional networks have achieved great success for visual recognition in still images. However, for action recognition in videos, the advantage over traditional methods is not so evident. This paper aims to discover the principles to design effective ConvNet architectures for action recognition in videos and learn these models given limited training samples. Our first contribution is temporal segment network (TSN), a novel framework for video-based action recognition. which is based on the idea of long-range temporal structure modeling. It combines a sparse temporal sampling strategy and video-level supervision to enable efficient and effective learning using the whole action video. The other contribution is our study on a series of good practices in learning ConvNets on video data with the help of temporal segment network. Our approach obtains the state-the-of-art performance on the datasets of HMDB51 ( $ 69.4\% $) and UCF101 ($ 94.2\% $). We also visualize the learned ConvNet models, which qualitatively demonstrates the effectiveness of temporal segment network and the proposed good practices.

Motivation & Objective

• Motivate effective video-level representations that capture long-range temporal structure in actions.
• Address limited training samples for deep ConvNets in video by proposing robust training practices.
• Develop a sparse, end-to-end framework that leverages whole-video information with efficient computation.

Proposed method

• Propose Temporal Segment Networks (TSN) that sample K short snippets sparsely from each video and aggregate snippet predictions with a segmental consensus to obtain a video-level prediction.
• Use a two-stream ConvNet architecture (spatial RGB stream and temporal flow stream) with very deep networks (BN-Inception) for each snippet.
• Explore multiple input modalities (RGB, RGB difference, optical flow, warped optical flow) and assess their complementary information.
• Apply cross-modality pre-training to initialize temporal networks from RGB-pretrained models, use partial Batch Normalization with dropout to mitigate overfitting, and implement data augmentation (corner cropping, scale jittering).
• Adopt a differentiable segmental aggregation (average pooling found effective) to fuse snippet-level predictions into a video-level score, enabling end-to-end training with backpropagation.

Experimental results

Research questions

• RQ1 How can long-range temporal structure in videos be effectively modeled for action recognition using deep ConvNets?
• RQ2 Can sparse temporal sampling plus segment-level aggregation achieve competitive performance with much lower computation than dense sampling?
• RQ3 What are the best practices to train very deep ConvNets on limited video data (pre-training, regularization, augmentation) and which input modalities maximize performance?

Key findings

• TSN with sparse sampling and segmental consensus yields state-of-the-art results on HMDB51 (69.4%) and UCF101 (94.2%) in their setup.
• Fusion of multiple modalities (RGB, RGB difference, optical flow, warped optical flow) improves recognition, with the best result using all four modalities achieving higher accuracy than any subset in their experiments.
• Cross-modality pre-training, partial BN with dropout, and advanced data augmentation substantially improve two-stream CNN performance under limited data.
• Average pooling as the segmental consensus function outperforms max and weighted average in their experiments.
• BN-Inception-based two-stream networks with TSN outperform other architectures, demonstrating that long-term temporal modeling is crucial for video action recognition.