[Paper Review] STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognition

Dasom Ahn, Sangwon Kim+2 more|arXiv (Cornell University)|Oct 14, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes STAR-Transformer, a novel spatio-temporal cross-attention transformer that jointly models video frames and human skeleton sequences via multi-class tokens for improved human action recognition. By integrating full, zigzag, and binary spatio-temporal attention mechanisms in an encoder-decoder architecture, STAR-Transformer achieves state-of-the-art performance on Penn-Action and NTU-RGB+D datasets, with a 98.7% accuracy using F-Z (encoder) and F-B (decoder) configurations.

ABSTRACT

In action recognition, although the combination of spatio-temporal videos and skeleton features can improve the recognition performance, a separate model and balancing feature representation for cross-modal data are required. To solve these problems, we propose Spatio-TemporAl cRoss (STAR)-transformer, which can effectively represent two cross-modal features as a recognizable vector. First, from the input video and skeleton sequence, video frames are output as global grid tokens and skeletons are output as joint map tokens, respectively. These tokens are then aggregated into multi-class tokens and input into STAR-transformer. The STAR-transformer encoder layer consists of a full self-attention (FAttn) module and a proposed zigzag spatio-temporal attention (ZAttn) module. Similarly, the continuous decoder consists of a FAttn module and a proposed binary spatio-temporal attention (BAttn) module. STAR-transformer learns an efficient multi-feature representation of the spatio-temporal features by properly arranging pairings of the FAttn, ZAttn, and BAttn modules. Experimental results on the Penn-Action, NTU RGB+D 60, and 120 datasets show that the proposed method achieves a promising improvement in performance in comparison to previous state-of-the-art methods.

Motivation & Objective

• To address the challenge of effectively combining video and skeleton features in action recognition without requiring separate cross-modal learning submodels.
• To overcome the limitations of standard Vision Transformers in modeling long-range temporal dependencies and multi-modal spatio-temporal relationships.
• To develop a unified, end-to-end trainable architecture that learns discriminative multi-feature representations from both RGB video and skeletal data.
• To improve recognition accuracy by enabling balanced attention across all temporal frames, avoiding over-reliance on final frames as seen in standard self-attention.
• To enable efficient, scalable feature aggregation through multi-class tokens derived from shared CNN-embedded video and skeleton inputs.

Proposed method

• The model uses a shared CNN to extract global grid tokens from video frames and joint map tokens from skeleton sequences, which are then aggregated into multi-class tokens for input to the STAR-Transformer.
• The encoder employs a full spatio-temporal attention (FAttn) and a novel zigzag spatio-temporal attention (ZAttn) module to capture long-range spatial and temporal dependencies across modalities.
• The decoder uses FAttn and a binary spatio-temporal attention (BAttn) module to reconstruct and refine the multi-feature representation for classification.
• The architecture is designed with an encoder-decoder structure where FAttn, ZAttn, and BAttn are strategically combined to balance global and local feature learning across time and space.
• Cross-attention mechanisms are specifically tailored to model interactions between video and skeleton modalities, enabling joint representation learning without separate feature fusion networks.
• The model is trained end-to-end using standard cross-entropy loss, with performance evaluated on standard benchmarks including Penn-Action and NTU-RGB+D 60/120.

Experimental results

Research questions

• RQ1 Can a unified transformer architecture effectively learn joint representations from both video frames and human skeleton sequences without requiring separate cross-modal learning modules?
• RQ2 How does the proposed zigzag and binary spatio-temporal attention mechanism improve temporal modeling compared to standard multi-head self-attention in Vision Transformers?
• RQ3 Does the use of multi-class tokens derived from shared CNN features enhance feature alignment and representation efficiency across modalities?
• RQ4 What is the optimal configuration of attention modules (FAttn, ZAttn, BAttn) in the encoder and decoder for maximizing action recognition accuracy?
• RQ5 To what extent does the proposed attention mechanism ensure balanced attention across all temporal frames, avoiding bias toward final frames?

Key findings

• The F-Z (encoder) and F-B (decoder) configuration achieved the highest accuracy of 98.7% on the NTU-RGB+D 120 dataset, outperforming previous state-of-the-art methods.
• The use of zigzag and binary spatio-temporal attention mechanisms led to more uniform attention scores across all frames, unlike standard FAttn, which heavily favored the final frames.
• When only FAttn was used, attention scores were concentrated on the last three frames of an action, indicating poor modeling of early and mid-sequence dynamics.
• The combination of ZAttn and BAttn in the encoder and decoder, respectively, produced more balanced and informative attention maps, especially during action transitions.
• The model achieved consistent performance gains across all three datasets—Penn-Action, NTU-RGB+D 60, and NTU-RGB+D 120—demonstrating robustness and generalization.
• The ablation study confirmed that the number of layers should be limited to three to prevent overfitting, with performance peaking at this depth.