[Paper Review] STH: Spatio-Temporal Hybrid Convolution for Efficient Action Recognition

Xu Li, Jingwen Wang+5 more|arXiv (Cornell University)|Mar 18, 2020

Human Pose and Action Recognition38 references4 citations

TL;DR

This paper proposes STH-Conv, a novel spatio-temporal hybrid convolution block that interleaves spatial and temporal kernels within a single 2D convolutional layer by grouping input channels, enabling deep integration of spatio-temporal features with minimal parameter cost. The method achieves state-of-the-art or competitive accuracy on Something-Something V1/V2, Jester, and HMDB-51 while maintaining model efficiency comparable to 2D CNNs and outperforming 3D CNNs in both accuracy and parameter efficiency.

ABSTRACT

Effective and Efficient spatio-temporal modeling is essential for action recognition. Existing methods suffer from the trade-off between model performance and model complexity. In this paper, we present a novel Spatio-Temporal Hybrid Convolution Network (denoted as "STH") which simultaneously encodes spatial and temporal video information with a small parameter cost. Different from existing works that sequentially or parallelly extract spatial and temporal information with different convolutional layers, we divide the input channels into multiple groups and interleave the spatial and temporal operations in one convolutional layer, which deeply incorporates spatial and temporal clues. Such a design enables efficient spatio-temporal modeling and maintains a small model scale. STH-Conv is a general building block, which can be plugged into existing 2D CNN architectures such as ResNet and MobileNet by replacing the conventional 2D-Conv blocks (2D convolutions). STH network achieves competitive or even better performance than its competitors on benchmark datasets such as Something-Something (V1 & V2), Jester, and HMDB-51. Moreover, STH enjoys performance superiority over 3D CNNs while maintaining an even smaller parameter cost than 2D CNNs.

Motivation & Objective

• To address the performance-complexity trade-off in existing spatio-temporal action recognition models.
• To enable deeper integration of spatial and temporal features within a single convolutional layer, avoiding sequential or parallel processing of spatial and temporal operations.
• To design a general-purpose building block that can be seamlessly integrated into existing 2D CNN architectures like ResNet and MobileNet.
• To achieve high accuracy on action recognition benchmarks while maintaining low parameter count and computational cost, suitable for mobile and embedded deployment.

Proposed method

• STH-Conv divides input channels into groups and interweaves basic spatial and temporal convolutional kernels along the channel dimension within a single 2D convolutional layer.
• The method constructs spatio-temporal hybrid kernels by arranging spatial and temporal kernels in a structured, interleaved manner, enabling joint modeling of appearance and motion.
• The STH block is designed as a plug-in replacement for standard 2D convolutional blocks in 2D CNNs, transforming them into efficient spatio-temporal networks.
• An adaptive attention mechanism is used to dynamically balance spatial and temporal components, with learned attention coefficients for each feature map.
• The network is trained end-to-end using standard cross-entropy loss, with no need for optical flow or additional supervision.
• The architecture supports plug-in deployment into existing 2D CNNs such as ResNet and MobileNet, enabling efficient video action recognition.

Experimental results

Research questions

• RQ1 Can a single convolutional layer effectively and efficiently model both spatial and temporal features in videos without increasing model complexity?
• RQ2 How does interleaving spatial and temporal kernels within a single 2D convolutional layer compare to sequential or parallel processing in terms of performance and efficiency?
• RQ3 To what extent can the STH block improve action recognition accuracy while maintaining a model size and FLOPs comparable to 2D CNNs?
• RQ4 How do the learned attention weights for spatial and temporal components vary across different datasets, network depths, and input modalities?

Key findings

• STH achieves a top-1 accuracy of 46.8% on Something-Something V1 with only 30.6 GFLOPs and 23.2M parameters, outperforming I3D and ECO while using fewer parameters.
• On Something-Something V2, STH achieves 78.5% top-1 accuracy with 61.3 GFLOPs and 23.2M parameters, surpassing TSM and ABM models in accuracy while maintaining lower complexity.
• The STH network learns to assign higher attention to temporal components in deeper layers, especially on temporal-rich datasets like Something-Something V1.
• For optical flow inputs, the network assigns significantly higher attention to temporal components, confirming its sensitivity to motion cues.
• The method achieves better performance than 3D CNNs (e.g., I3D) while using fewer parameters and FLOPs, demonstrating superior efficiency-accuracy trade-off.
• The plug-in nature of STH allows it to be seamlessly integrated into existing 2D CNNs like ResNet and MobileNet, enabling efficient video action recognition without architectural overhaul.