[Paper Review] Convolutional Two-Stream Network Fusion for Video Action Recognition

Christoph Feichtenhofer, Axel Pinz+1 more|arXiv (Cornell University)|Apr 22, 2016

Human Pose and Action Recognition31 references371 citations

TL;DR

This paper presents a spatiotemporal fusion architecture that combines spatial and temporal ConvNet streams using various fusion strategies, showing state-of-the-art results on UCF101 and HMDB51. It demonstrates that late-stage spatial fusion with 3D temporal pooling yields strong performance with fewer parameters than simple softmax fusion.

ABSTRACT

Recent applications of Convolutional Neural Networks (ConvNets) for human action recognition in videos have proposed different solutions for incorporating the appearance and motion information. We study a number of ways of fusing ConvNet towers both spatially and temporally in order to best take advantage of this spatio-temporal information. We make the following findings: (i) that rather than fusing at the softmax layer, a spatial and temporal network can be fused at a convolution layer without loss of performance, but with a substantial saving in parameters; (ii) that it is better to fuse such networks spatially at the last convolutional layer than earlier, and that additionally fusing at the class prediction layer can boost accuracy; finally (iii) that pooling of abstract convolutional features over spatiotemporal neighbourhoods further boosts performance. Based on these studies we propose a new ConvNet architecture for spatiotemporal fusion of video snippets, and evaluate its performance on standard benchmarks where this architecture achieves state-of-the-art results.

Motivation & Objective

• Motivate and improve action recognition by effectively fusing appearance (spatial) and motion (temporal) cues from video.
• Investigate where, how, and how often to fuse two ConvNet streams to maximize spatiotemporal feature learning.
• Develop a practical spatiotemporal fusion architecture that preserves spatial correspondence while leveraging temporal context.
• Compare fusion strategies and depths to understand their impact on accuracy and model size.

Proposed method

• Evaluate multiple fusion functions (sum, max, concatenation, conv, bilinear) to combine two streams at chosen layers.
• Experiment with fusion locations (after various conv layers, FC layers, or multi-layer fusion) under the constraint of matching spatial dimensions.
• Implement temporal fusion via 2D/3D pooling and 3D convolutions to capture short-term and long-term temporal structure.
• Propose a spatiotemporal fusion architecture that fuses at the last convolutional layer with 3D conv fusion and 3D pooling, while preserving the temporal stream.
• Train two-stream networks (spatial: RGB, temporal: optical flow) pretrained on ImageNet, then finetune on UCF101 and HMDB51; evaluate with dense temporal sampling at test time.

Experimental results

Research questions

• RQ1 What fusion strategy between spatial and temporal streams yields the best action recognition accuracy?
• RQ2 Where in the network should fusion occur to maximize performance while minimizing parameters?
• RQ3 How should temporal information be fused to capture short-term and long-term dynamics effectively?
• RQ4 Does using deeper networks (e.g., VGG-16) improve action recognition more than deeper temporal models?
• RQ5 How does spatiotemporal fusion compare to single-stream or late fusion baselines on standard benchmarks?

Key findings

• Conv fusion at the last convolutional layer (ReLU5) outperforms other spatial fusion layers and is competitive with or better than late fusion at the softmax layer, with substantially fewer parameters.
• Concatenation and max fusion generally underperform compared to sum or conv fusion for spatial fusion, and conv fusion provides the best accuracy in many settings.
• Fusing the two streams at ReLU5 and using 3D fusion then 3D pooling improves performance versus 2D pooling, and keeps the architectural benefits of explicit spatiotemporal correspondence.
• In deeper models (VGG-16) for both streams, accuracy improves notably for spatial models, while temporal gains are smaller, indicating stronger benefits from spatial depth.
• Temporal fusion using 3D convs and 3D pooling yields higher accuracy than plain 2D fusion or simple averaging of predictions; a 3D fusion filter further boosts performance on benchmarks.
• The proposed 3D spatiotemporal fusion architecture achieves state-of-the-art results on UCF101 and HMDB51 compared with prior two-stream methods.