[Paper Review] STM: SpatioTemporal and Motion Encoding for Action Recognition

Boyuan Jiang, Mengmeng Wang+3 more|arXiv (Cornell University)|Aug 7, 2019

Human Pose and Action RecognitionComputer Science39 references59 citations

TL;DR

STM proposes a unified 2D CNN that encodes spatiotemporal and motion features via Channel-wise SpatioTemporal Module (CSTM) and Channel-wise Motion Module (CMM), replacing residual blocks in ResNet to outperform state-of-the-art without 3D convolutions or optical flow precomputation.

ABSTRACT

Spatiotemporal and motion features are two complementary and crucial information for video action recognition. Recent state-of-the-art methods adopt a 3D CNN stream to learn spatiotemporal features and another flow stream to learn motion features. In this work, we aim to efficiently encode these two features in a unified 2D framework. To this end, we first propose an STM block, which contains a Channel-wise SpatioTemporal Module (CSTM) to present the spatiotemporal features and a Channel-wise Motion Module (CMM) to efficiently encode motion features. We then replace original residual blocks in the ResNet architecture with STM blcoks to form a simple yet effective STM network by introducing very limited extra computation cost. Extensive experiments demonstrate that the proposed STM network outperforms the state-of-the-art methods on both temporal-related datasets (i.e., Something-Something v1 & v2 and Jester) and scene-related datasets (i.e., Kinetics-400, UCF-101, and HMDB-51) with the help of encoding spatiotemporal and motion features together.

Motivation & Objective

• Motivate efficient action recognition by jointly encoding spatiotemporal and motion information.
• Eliminate the need for 3D convolutions and optical flow precomputation while maintaining high accuracy.
• Enable easy integration into existing 2D CNN backbones (e.g., ResNet) with minimal computational overhead.

Proposed method

• Introduce Channel-wise SpatioTemporal Module (CSTM) to fuse temporal information per channel with local spatial processing.
• Introduce Channel-wise Motion Module (CMM) to extract motion-like representations from consecutive frames without explicit optical flow.
• Form STM block by combining CSTM and CMM with a residual-style identity path and a 1x1 channel dimension reduction/expansion, enabling insertion into ResNet architectures.
• Replace standard residual blocks in ResNet with STM blocks to build the STM network, achieving a lightweight 2D CNN without 3D ops or precomputed flow.
• Demonstrate that summation fusion of CSTM and CMM outperforms concatenation, and that placing STM blocks deeper in the network yields better results.

Experimental results

Research questions

• RQ1 Can spatiotemporal and motion features be effectively encoded in a unified 2D CNN without 3D convolutions or optical flow precomputation?
• RQ2 Do channel-wise (per-channel) temporal fusion and channel-wise motion representations provide complementary gains for action recognition?
• RQ3 What is the impact of fusion strategy, block placement, and temporal kernel design on STM performance?

Key findings

• STM achieves state-of-the-art results on temporal-related datasets (Something-Something v1/v2, Jester) using only RGB frames.
• On Something-Something v1 with 8 frames, STM improves top-1 accuracy by about 29.5% over TSN; with 16 frames, improvements reach 34.5–34.2% over TSN.
• On Something-Something v2, STM yields substantial gains over TSN with both 8 and 16 frames (top-1 and top-5 metrics).
• On Kinetics-400, STM achieves 73.7% top-1 and 91.6% top-5, competitive with 3D and two-stream methods while being lighter in computation.
• On UCF-101 and HMDB-51, STM approaches or surpasses several baselines, outperforming RGB-only TSN and competing with I3D-based approaches, especially when pretraining on Kinetics is used.
• Ablation studies show: CSTM and CMM each contribute substantial gains, with channel-wise temporal fusion outperforming ordinary temporal convolution, and summation fusion outperforming concatenation.