[Paper Review] Motion Feature Network: Fixed Motion Filter for Action Recognition

Myunggi Lee, Seungeui Lee+3 more|arXiv (Cornell University)|Jul 26, 2018

Human Pose and Action Recognition24 references4 citations

TL;DR

This paper proposes MFNet, a unified 2D CNN architecture with learnable motion blocks that encode spatio-temporal features directly from RGB frames using fixed directional filters, eliminating the need for optical flow or 3D convolutions. The method achieves state-of-the-art performance on Jester (96.22% top-1 accuracy) and Something-Something (37.48% top-1 accuracy) by training from scratch without pre-training or multi-stream fusion.

ABSTRACT

Spatio-temporal representations in frame sequences play an important role in the task of action recognition. Previously, a method of using optical flow as a temporal information in combination with a set of RGB images that contain spatial information has shown great performance enhancement in the action recognition tasks. However, it has an expensive computational cost and requires two-stream (RGB and optical flow) framework. In this paper, we propose MFNet (Motion Feature Network) containing motion blocks which make it possible to encode spatio-temporal information between adjacent frames in a unified network that can be trained end-to-end. The motion block can be attached to any existing CNN-based action recognition frameworks with only a small additional cost. We evaluated our network on two of the action recognition datasets (Jester and Something-Something) and achieved competitive performances for both datasets by training the networks from scratch.

Motivation & Objective

• To address the high computational cost and complexity of two-stream and 3D CNN methods in action recognition.
• To enable end-to-end learning of spatio-temporal features using only RGB input, avoiding pre-computed optical flow.
• To design a lightweight, modular motion block that can be integrated into existing 2D CNN frameworks with minimal overhead.
• To improve performance on action recognition datasets with fine-grained, temporally symmetric actions where spatial cues alone are insufficient.
• To validate the effectiveness of motion blocks in capturing motion direction and temporal dynamics without additional modality supervision.

Proposed method

• Introduces motion blocks that apply fixed directional filters (e.g., horizontal, vertical, diagonal) to feature maps from consecutive RGB frames.
• Computes motion features by subtracting the shifted feature map from the original, mimicking optical flow but without learning parameters.
• Integrates motion blocks into a 2D CNN backbone (e.g., ResNet) as a plug-in module, enabling end-to-end training.
• Uses a two-stream-like inference strategy with segment-wise feature aggregation, but only on RGB inputs.
• Employs a shared feature extractor for spatial features and a separate motion block for temporal modeling, both trained jointly.
• Applies temporal average pooling over segments to aggregate features, enabling classification with minimal architectural changes.

Experimental results

Research questions

• RQ1 Can a fixed, non-learned motion filter effectively capture spatio-temporal dynamics in video action recognition?
• RQ2 Does a unified 2D CNN with motion blocks outperform two-stream or 3D CNN baselines when trained from scratch on RGB-only inputs?
• RQ3 How does the proposed motion block improve classification accuracy on datasets with temporally symmetric action pairs, such as 'Swiping Left' vs 'Swiping Right'?
• RQ4 What is the optimal number of segments for inference, and how does it relate to the number used during training?
• RQ5 Can the motion block be effectively transferred or generalized across different action recognition datasets without pre-training on large-scale datasets?

Key findings

• MFNet achieves 96.22% top-1 accuracy on the Jester test set, outperforming previous methods without using optical flow or pre-trained ImageNet weights.
• On the Something-Something dataset, MFNet reaches 37.48% top-1 accuracy on the official leaderboard, demonstrating strong performance on fine-grained action recognition.
• The model significantly reduces misclassification of temporally symmetric action pairs—e.g., only 4.2% of 'Rolling Hand Forward' predictions were misclassified as 'Rolling Hand Backward', compared to 35.7% in the baseline.
• Performance improves with more validation segments, peaking at K=7, though the optimal number is slightly higher than the training K, indicating benefit from finer temporal sampling.
• The ablation study confirms that the motion block contributes substantially to performance, especially in distinguishing motion direction, even without any pre-training or multi-scale fusion.
• MFNet achieves competitive results with fewer parameters and lower computational cost than 3D ConvNets and two-stream models, due to its 2D CNN foundation and fixed filters.