[Paper Review] Temporal Segment Networks for Action Recognition in Videos

Limin Wang, Yuanjun Xiong+5 more|arXiv (Cornell University)|May 8, 2017

Human Pose and Action RecognitionComputer Science59 references53 citations

TL;DR

Introduces Temporal Segment Networks (TSN) to model long-range temporal structure in videos via sparse segment sampling and segmental consensus, achieving state-of-the-art on multiple action recognition benchmarks and enabling real-time RGB-diff based motion.

ABSTRACT

Deep convolutional networks have achieved great success for image recognition. However, for action recognition in videos, their advantage over traditional methods is not so evident. We present a general and flexible video-level framework for learning action models in videos. This method, called temporal segment network (TSN), aims to model long-range temporal structures with a new segment-based sampling and aggregation module. This unique design enables our TSN to efficiently learn action models by using the whole action videos. The learned models could be easily adapted for action recognition in both trimmed and untrimmed videos with simple average pooling and multi-scale temporal window integration, respectively. We also study a series of good practices for the instantiation of TSN framework given limited training samples. Our approach obtains the state-the-of-art performance on four challenging action recognition benchmarks: HMDB51 (71.0%), UCF101 (94.9%), THUMOS14 (80.1%), and ActivityNet v1.2 (89.6%). Using the proposed RGB difference for motion models, our method can still achieve competitive accuracy on UCF101 (91.0%) while running at 340 FPS. Furthermore, based on the temporal segment networks, we won the video classification track at the ActivityNet challenge 2016 among 24 teams, which demonstrates the effectiveness of TSN and the proposed good practices.

Motivation & Objective

• Model long-range temporal structure in videos for action recognition.
• Develop a video-level framework that uses sparse sampling to handle full videos.
• Enable TSN to work with trimmed and untrimmed videos through hierarchical aggregation.
• Identify good practices for training deep action models with limited data.

Proposed method

• Divide a video into K segments and sample one snippet per segment.
• Process each snippet with a shared ConvNet to obtain snippet scores.
• Aggregate snippet scores with flexible consensus functions (max, average, top-K, weighted, attention).
• Use Multi-scale Temporal Window Integration (M-TWI) to apply models to untrimmed videos.
• Explore cross-modality initialization and partial batch normalization to improve training with limited data.
• Experiment with modalities including RGB, optical flow, RGB difference, and warped flow.

Experimental results

Research questions

• RQ1 How can long-range temporal structure be effectively captured for action recognition in videos with a lightweight sampling strategy?
• RQ2 Can a segment-based aggregation framework enable accurate recognition in both trimmed and untrimmed videos?
• RQ3 What input modalities and training practices best improve performance under limited data?
• RQ4 How do different aggregation strategies affect video-level predictions and training dynamics?
• RQ5 What is the impact of cross-modality initialization and partial BN on model performance?

Key findings

• Achieved state-of-the-art accuracy on HMDB51 (71.0%), UCF101 (94.9%), THUMOS14 (80.1%), and ActivityNet v1.2 (89.6%).
• RGB-difference as motion input can reach 91.0% on UCF101 while running at 340 FPS.
• The framework supports trimming flexibility and achieves strong results with multi-scale temporal window integration for untrimmed videos.
• Introduced five aggregation functions and showed that Top-K pooling and attention weighting improve robustness to background.
• Validated that cross-modality initialization and partial BN help train deep models with limited data on action recognition tasks.