[Paper Review] Cross-Modal Learning with 3D Deformable Attention for Action Recognition

Sangwon Kim, Dasom Ahn+1 more|arXiv (Cornell University)|Dec 12, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes a 3D deformable transformer with cross-modal learning for action recognition, using adaptive spatiotemporal attention to focus on relevant joints and temporal segments. It achieves state-of-the-art performance on NTU60, NTU120, FineGYM, and PennAction without pre-training, while enabling explainable attention via joint and temporal stride visualization.

ABSTRACT

An important challenge in vision-based action recognition is the embedding of spatiotemporal features with two or more heterogeneous modalities into a single feature. In this study, we propose a new 3D deformable transformer for action recognition with adaptive spatiotemporal receptive fields and a cross-modal learning scheme. The 3D deformable transformer consists of three attention modules: 3D deformability, local joint stride, and temporal stride attention. The two cross-modal tokens are input into the 3D deformable attention module to create a cross-attention token with a reflected spatiotemporal correlation. Local joint stride attention is applied to spatially combine attention and pose tokens. Temporal stride attention temporally reduces the number of input tokens in the attention module and supports temporal expression learning without the simultaneous use of all tokens. The deformable transformer iterates L-times and combines the last cross-modal token for classification. The proposed 3D deformable transformer was tested on the NTU60, NTU120, FineGYM, and PennAction datasets, and showed results better than or similar to pre-trained state-of-the-art methods even without a pre-training process. In addition, by visualizing important joints and correlations during action recognition through spatial joint and temporal stride attention, the possibility of achieving an explainable potential for action recognition is presented.

Motivation & Objective

• To address the limitation of fixed receptive fields in vision transformers for spatiotemporal action recognition.
• To enable effective cross-modal learning between video and skeleton modalities within a single transformer architecture.
• To reduce computational complexity while preserving temporal and spatial correlations through adaptive attention mechanisms.
• To provide explainable attention maps by visualizing key joints and temporal segments during action recognition.

Proposed method

• Introduces a 3D deformable attention module that adaptively selects relevant spatiotemporal tokens instead of attending to all tokens, reducing computational cost and improving feature focus.
• Employs a cross-modal token mechanism that enables bidirectional contextual exchange between video and pose modalities within a single transformer block.
• Applies local joint stride attention by grouping joint tokens with a sliding window to maintain spatial correlation while improving efficiency.
• Uses temporal stride attention to downsample input tokens over time, preserving long-range temporal dependencies with reduced sequence length.
• Combines three attention modules—3D deformable, local joint stride, and temporal stride—into a single iterative transformer block for end-to-end feature learning.
• Projects skeleton data into joint tokens and fuses them with video features through cross-attention, eliminating the need for auxiliary submodels.

Experimental results

Research questions

• RQ1 Can a 3D deformable attention mechanism improve spatiotemporal feature learning in vision transformers for action recognition by focusing on relevant regions adaptively?
• RQ2 How effective is a unified transformer architecture with cross-modal tokens for fusing video and skeleton modalities without separate encoders?
• RQ3 To what extent does local joint stride and temporal stride attention improve efficiency and performance while maintaining spatial-temporal correlation?
• RQ4 Can the proposed attention mechanism produce explainable attention maps that align with human perception of key action components?
• RQ5 What is the optimal number of input frames and attention window configuration for balancing accuracy and computational cost?

Key findings

• The proposed 3D deformable attention achieved a top-1 accuracy of 99.7% on the PennAction dataset, outperforming models without this component.
• Ablation studies confirmed that removing 3D deformable attention reduced accuracy by 4.9 percentage points, demonstrating its critical role in performance.
• Using cross-modal tokens improved accuracy from 92.2% (no tokens) to 99.7%, proving their effectiveness in cross-modal feature fusion.
• The optimal number of input frames for the PennAction dataset was found to be 12, as performance dropped when fewer frames were used.
• Temporal stride attention with a stride of half the window size achieved the best performance, indicating that moderate overlap enhances temporal correlation learning.
• Visualization showed that attention was strongly activated on moving joints (e.g., head, right hand, right leg), confirming alignment with action dynamics and supporting model interpretability.