[Paper Review] Two-Stream Convolutional Networks for Action Recognition in Videos

Karen Simonyan, Andrew Zisserman|arXiv (Cornell University)|Jun 9, 2014

Human Pose and Action Recognition30 references5,353 citations

TL;DR

The paper presents a two-stream CNN architecture for video action recognition, combining a spatial stream (frame appearance) and a temporal stream (dense optical flow), with multi-task learning and fusion strategies improving state-of-the-art performance on UCF-101 and HMDB-51.

ABSTRACT

We investigate architectures of discriminatively trained deep Convolutional Networks (ConvNets) for action recognition in video. The challenge is to capture the complementary information on appearance from still frames and motion between frames. We also aim to generalise the best performing hand-crafted features within a data-driven learning framework. Our contribution is three-fold. First, we propose a two-stream ConvNet architecture which incorporates spatial and temporal networks. Second, we demonstrate that a ConvNet trained on multi-frame dense optical flow is able to achieve very good performance in spite of limited training data. Finally, we show that multi-task learning, applied to two different action classification datasets, can be used to increase the amount of training data and improve the performance on both. Our architecture is trained and evaluated on the standard video actions benchmarks of UCF-101 and HMDB-51, where it is competitive with the state of the art. It also exceeds by a large margin previous attempts to use deep nets for video classification.

Motivation & Objective

• Motivate deep learning for video action recognition by leveraging complementary appearance and motion information.
• Propose a two-stream CNN architecture with separate spatial and temporal networks and late fusion for action classification.
• Investigate optical-flow–based temporal input configurations and their impact on recognition performance.
• Explore multi-task learning to fuse training data from multiple datasets and improve generalization.

Proposed method

• Introduce a two-stream ConvNet architecture with spatial (frame-based) and temporal (motion-based) streams.
• Pre-train the spatial stream on ImageNet and fine-tune for action recognition; use late fusion to combine stream scores.
• Describe the temporal ConvNet that operates on multi-frame dense optical flow inputs (two options: optical flow stacking and trajectory stacking).
• Experiment with mean flow subtraction and bi-directional flow variants to assess robustness to camera motion and motion patterns.
• Apply multi-task learning by jointly training on HMDB-51 and UCF-101 with task-specific softmax layers and losses.
• Evaluate using mean accuracy across three splits on UCF-101 and HMDB-51 and compare against state-of-the-art hand-crafted and deep methods.

Experimental results

Research questions

• RQ1 Can a dual-stream CNN leveraging both appearance and motion signals outperform single-stream approaches in video action recognition?
• RQ2 How does multi-frame optical flow input (and its variants) affect temporal feature learning and accuracy?
• RQ3 Does multi-task learning across datasets improve temporal CNN generalization for action recognition?

Key findings

• The two-stream architecture improves performance by combining complementary spatial and temporal cues.
• Temporal nets trained on optical-flow inputs with multi-frame stacking achieve substantially higher accuracy than single-frame or spatial nets.
• Mean subtraction (camera-motion compensation) and L=10 optical-flow stacking yield strong temporal performance improvements.
• Multi-task learning using HMDB-51 and UCF-101 provides the best temporal-net results, showing benefits from additional training data.
• Two-stream fusion with SVM surpasses averaging fusion, with temporal stream trained via multi-task learning achieving the best overall results.
• On UCF-101, the two-stream model with SVM fusion reaches 88.0% mean accuracy (split 1), and on HMDB-51 it reaches 59.4%.