[Paper Review] Residual Frames with Efficient Pseudo-3D CNN for Human Action Recognition

Jiawei Chen, Jenson Hsiao+1 more|arXiv (Cornell University)|Aug 3, 2020

Human Pose and Action Recognition12 references4 citations

TL;DR

This paper proposes a lightweight, efficient framework for human action recognition by leveraging residual frames—differences between adjacent RGB frames—as a motion representation to capture salient motion cues with minimal computational cost. It introduces a novel pseudo-3D convolution module that decouples 3D convolutions into 2D and 1D operations, incorporates feature-space residuals and a self-attention mechanism, achieving state-of-the-art accuracy with 83% top-1 accuracy on UCF101 while reducing FLOPs from 163G to 30G.

ABSTRACT

Human action recognition is regarded as a key cornerstone in domains such as surveillance or video understanding. Despite recent progress in the development of end-to-end solutions for video-based action recognition, achieving state-of-the-art performance still requires using auxiliary hand-crafted motion representations, e.g., optical flow, which are usually computationally demanding. In this work, we propose to use residual frames (i.e., differences between adjacent RGB frames) as an alternative "lightweight" motion representation, which carries salient motion information and is computationally efficient. In addition, we develop a new pseudo-3D convolution module which decouples 3D convolution into 2D and 1D convolution. The proposed module exploits residual information in the feature space to better structure motions, and is equipped with a self-attention mechanism that assists to recalibrate the appearance and motion features. Empirical results confirm the efficiency and effectiveness of residual frames as well as the proposed pseudo-3D convolution module.

Motivation & Objective

• To improve human action recognition performance using a computationally efficient motion representation alternative to optical flow.
• To address the challenge of modeling temporal dynamics in video data without relying on expensive hand-crafted motion features.
• To develop a lightweight, parameter-efficient 3D convolution module that enhances motion and appearance feature learning.
• To validate the effectiveness of residual frames as a complementary modality to RGB input for action recognition.
• To demonstrate that feature-space residuals and self-attention can significantly improve recognition performance.

Proposed method

• Residual frames are computed as the absolute difference between adjacent RGB frames, i.e., $ x^{res}_{(t_1,s)} = |x_{t_1+s} - x_{t_1}| $, to extract motion-specific features while minimizing background and appearance noise.
• A new pseudo-3D convolution module decouples standard 3D convolution into parallel 2D spatial and 1D temporal convolutions to reduce computational cost and model size.
• The module incorporates residuals from temporally adjacent feature maps to enhance motion representation learning in the feature space.
• A self-attention mechanism is introduced to recalibrate the importance of appearance and motion features based on their relevance to the action recognition task.
• The model is trained end-to-end using both RGB and residual frame inputs, with ablation studies to validate component contributions.
• The framework is evaluated on UCF101 using top-1 and top-5 accuracy, FLOPs, and ablation analysis to assess modality fusion and component effectiveness.

Experimental results

Research questions

• RQ1 Can residual frames, computed as frame differences, serve as an effective and efficient alternative to optical flow for human action recognition?
• RQ2 How does the performance of action recognition vary with different step sizes in residual frame computation?
• RQ3 To what extent does the proposed pseudo-3D convolution module improve recognition accuracy while reducing computational cost?
• RQ4 What is the individual contribution of the self-attention mechanism and feature-space residuals in the proposed module?
• RQ5 Does fusing RGB and residual frame inputs lead to better performance than using either modality alone?

Key findings

• Using only residual frames with step size $ s=1 $ achieves 83.0% top-1 accuracy on UCF101, outperforming RGB-only models by 2.6 percentage points.
• Fusing RGB and residual frames (with $ s=1 $) increases top-1 accuracy to 85.6%, demonstrating complementary information between modalities.
• The proposed pseudo-3D module reduces FLOPs from 163G (standard 3D) to 30G while improving performance, confirming its efficiency and effectiveness.
• Ablation studies show that removing the self-attention mechanism reduces top-1 accuracy by 1.8% (to 83.8%), and removing feature-space residuals reduces it by 2.1% (to 83.5%).
• The performance degrades with larger step sizes: top-1 accuracy drops from 83.0% ($ s=1 $) to 80.2% ($ s=4 $), indicating that small temporal steps preserve motion fidelity better.
• The full model with both self-attention and feature-space residuals achieves 85.6% top-1 accuracy, confirming the synergistic benefit of both components.