[Paper Review] Depthwise Spatio-Temporal STFT Convolutional Neural Networks for Human Action Recognition

Sudhakar Kumawat, Manisha Verma+2 more|arXiv (Cornell University)|Jul 22, 2020

Human Pose and Action RecognitionComputer Science75 references56 citations

TL;DR

This paper proposes spatio-temporal Short-Time Fourier Transform (STFT) blocks as a parameter-efficient alternative to 3D convolutional layers in 3D CNNs for human action recognition. By using non-trainable STFT kernels to extract local Fourier features in space and time, followed by trainable linear weights for channel correlation, the method reduces parameters by 3.5–4.5× and computation by 1.5–1.8× while achieving state-of-the-art or superior accuracy on seven benchmarks including Kinetics-400 and Jester.

ABSTRACT

Conventional 3D convolutional neural networks (CNNs) are computationally expensive, memory intensive, prone to overfitting, and most importantly, there is a need to improve their feature learning capabilities. To address these issues, we propose spatio-temporal short term Fourier transform (STFT) blocks, a new class of convolutional blocks that can serve as an alternative to the 3D convolutional layer and its variants in 3D CNNs. An STFT block consists of non-trainable convolution layers that capture spatially and/or temporally local Fourier information using a STFT kernel at multiple low frequency points, followed by a set of trainable linear weights for learning channel correlations. The STFT blocks significantly reduce the space-time complexity in 3D CNNs. In general, they use 3.5 to 4.5 times less parameters and 1.5 to 1.8 times less computational costs when compared to the state-of-the-art methods. Furthermore, their feature learning capabilities are significantly better than the conventional 3D convolutional layer and its variants. Our extensive evaluation on seven action recognition datasets, including Something-something v1 and v2, Jester, Diving-48, Kinetics-400, UCF 101, and HMDB 51, demonstrate that STFT blocks based 3D CNNs achieve on par or even better performance compared to the state-of-the-art methods.

Motivation & Objective

• To address the high computational cost, memory usage, and overfitting issues in 3D CNNs for video action recognition.
• To improve the feature learning capability of 3D CNNs by replacing conventional 3D convolutions with a new STFT-based block architecture.
• To design a parameter- and computation-efficient architecture that maintains or exceeds state-of-the-art performance on multiple action recognition datasets.
• To explore the effectiveness of STFT-based feature extraction across different spatio-temporal dimensions in video modeling.

Proposed method

• Proposes three variants of STFT blocks: ST-STFT (3D STFT), S-STFT (2D spatial STFT + depthwise temporal), and T-STFT (depthwise spatial + 1D temporal STFT).
• Employs non-trainable STFT kernels to extract local Fourier coefficients at multiple low-frequency points in spatio-temporal volumes.
• Uses depthwise 3D convolutions to process STFT outputs, enabling efficient computation across spatial and temporal dimensions.
• Introduces trainable linear weights after the STFT layer to model inter-channel correlations and enhance feature representation.
• Applies the STFT blocks within a bottleneck architecture, replacing standard 3D convolutions in ResNet-style networks.
• Trains models from scratch on large datasets like Kinetics-400, avoiding reliance on ImageNet pre-training to evaluate generalization.

Experimental results

Research questions

• RQ1 Can STFT-based blocks reduce the number of parameters and FLOPs in 3D CNNs without sacrificing performance?
• RQ2 How do STFT blocks compare to conventional 3D convolutions and factorized variants (e.g., R(2+1)D, S3D) in learning spatio-temporal features?
• RQ3 Does the use of non-trainable STFT kernels improve feature representation compared to learnable 3D convolutions?
• RQ4 How does the performance of STFT-based models scale across datasets of varying complexity and size, such as Jester, Kinetics-400, and UCF-101?
• RQ5 Can STFT-based models generalize well to small-scale datasets when pre-trained on large datasets like Kinetics?

Key findings

• The T-STFT variant achieved 96.94% accuracy on the Jester dataset, outperforming the SOTA STM model by 0.24% while using 1.6× fewer FLOPs and 4.1× fewer parameters.
• On Kinetics-400, the T-STFT model achieved 61.1% top-1 accuracy, outperforming the R(2+1)D-ResNet18 baseline (56.8%) despite using 5.2× fewer parameters.
• The ST-STFT block achieved 39.4% top-1 accuracy on Kinetics-400, demonstrating strong performance with minimal parameters.
• On UCF-101 and HMDB-51, Kinetics-pretrained STFT models achieved 94.7% and 71.5% accuracy respectively, comparable to ImageNet+Kinetics pre-trained SOTA models.
• The STFT-based models showed improved robustness to overfitting due to significantly reduced parameter counts (3–5× less than SOTA), even when trained from scratch.
• The method achieved consistent performance gains across diverse datasets, including temporally challenging ones like Jester and large-scale scene datasets like Kinetics-400.