[Paper Review] Learning Efficient Video Representation with Video Shuffle Networks

Pingchuan Ma, Yao Zhou+2 more|arXiv (Cornell University)|Nov 26, 2019

Human Pose and Action Recognition48 references4 citations

TL;DR

This paper proposes Video Shuffle Networks (VSN), a parameter-free, plug-in module that enables standard 2D convolutional networks to learn spatiotemporal representations by reorganizing feature maps across frames via channel-wise grouping and temporal shuffling. The method achieves state-of-the-art performance on Something-Something-V1/V2 and Moments in Time with zero additional parameters or FLOPs, significantly improving accuracy over 2D CNN baselines while maintaining low inference latency.

ABSTRACT

3D CNN shows its strong ability in learning spatiotemporal representation in recent video recognition tasks. However, inflating 2D convolution to 3D inevitably introduces additional computational costs, making it cumbersome in practical deployment. We consider whether there is a way to equip the conventional 2D convolution with temporal vision no requiring expanding its kernel. To this end, we propose the video shuffle, a parameter-free plug-in component that efficiently reallocates the inputs of 2D convolution so that its receptive field can be extended to the temporal dimension. In practical, video shuffle firstly divides each frame feature into multiple groups and then aggregate the grouped features via temporal shuffle operation. This allows the following 2D convolution aggregate the global spatiotemporal features. The proposed video shuffle can be flexibly inserted into popular 2D CNNs, forming the Video Shuffle Networks (VSN). With a simple yet efficient implementation, VSN performs surprisingly well on temporal modeling benchmarks. In experiments, VSN not only gains non-trivial improvements on Kinetics and Moments in Time, but also achieves state-of-the-art performance on Something-Something-V1, Something-Something-V2 datasets.

Motivation & Objective

• To address the high computational cost of 3D CNNs in video recognition while preserving strong temporal modeling capability.
• To enable standard 2D CNNs to learn temporal dependencies without expanding kernel size or adding learnable parameters.
• To design a lightweight, generic, and efficient component that can be seamlessly plugged into existing 2D video models.
• To improve performance on long-range and fine-grained action recognition benchmarks without increasing inference latency.

Proposed method

• Introduce video shuffle: a parameter-free operation that divides each frame’s feature channels into equal-sized groups and reorganizes them across temporal frames by stacking grouped features with the same index along the temporal dimension.
• Apply video shuffle before and after 2D convolutions in residual blocks to allow 2D convolutions to access global spatiotemporal context.
• Use an inverse shuffle operation to restore original spatial feature layout after temporal modeling, ensuring compatibility with standard 2D CNN architectures.
• Integrate video shuffle into the TSN framework using ResNet-50 and ResNet-101 backbones as the base model.
• Design two variants of residual blocks—headtail and compact—where video shuffle is inserted at specific positions to optimize temporal modeling.
• Train end-to-end with RGB inputs and evaluate on Kinetics, Moments in Time, and Something-Something datasets.

Experimental results

Research questions

• RQ1 Can 2D convolutions be effectively extended to model temporal dependencies without increasing parameters or FLOPs?
• RQ2 How does the proposed video shuffle operation compare to existing temporal modeling modules like temporal shift or 3D convolutions in performance and efficiency?
• RQ3 What is the optimal placement and number of video shuffle blocks within a 2D CNN for temporal modeling?
• RQ4 Can video shuffle achieve state-of-the-art performance on challenging temporal-action recognition benchmarks like Something-Something-V1/V2?

Key findings

• VSN-R50 achieves 44.5% top-1 accuracy on Something-Something-V2, surpassing the previous SOTA by 2.1% with no additional parameters or FLOPs.
• VSN-R101 achieves 46.5% accuracy on Something-Something-V2, outperforming the baseline TSN-ResNet-101 by 23.8 percentage points.
• On Kinetics, VSN-R50 improves accuracy by 2.0% over the baseline TSN-ResNet-50, and VSN-R101 achieves 75.4% accuracy, a 2.6% gain over the baseline.
• VSN maintains low latency—16.5ms for VSN-R50 and 28.7ms for VSN-R101—achieving 79.5 and 47.2 videos per second, respectively, outperforming I3D by 13× in speed.
• The compact residual block variant outperforms the headtail variant by 2.0% on Kinetics and 26.4% on Something-Something-V1, indicating superior design choice.
• Combining video shuffle with temporal shift modules yields the highest accuracy (73.5% on Kinetics, 46.6% on Sth-V1), demonstrating complementary strengths.