[Paper Review] Learning Discriminative Representations for Skeleton Based Action Recognition

Huanyu Zhou, Qingjie Liu+1 more|arXiv (Cornell University)|Mar 7, 2023

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes a plug-and-play feature refinement head (FR-Head) that enhances discriminative representations in skeleton-based action recognition by leveraging spatial-temporal decoupling and contrastive learning. It dynamically identifies and calibrates ambiguous samples using confident prototypes, achieving state-of-the-art performance on NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets, with gains of up to 1.4% in top-1 accuracy on ambiguous action groups.

ABSTRACT

Human action recognition aims at classifying the category of human action from a segment of a video. Recently, people have dived into designing GCN-based models to extract features from skeletons for performing this task, because skeleton representations are much more efficient and robust than other modalities such as RGB frames. However, when employing the skeleton data, some important clues like related items are also discarded. It results in some ambiguous actions that are hard to be distinguished and tend to be misclassified. To alleviate this problem, we propose an auxiliary feature refinement head (FR Head), which consists of spatial-temporal decoupling and contrastive feature refinement, to obtain discriminative representations of skeletons. Ambiguous samples are dynamically discovered and calibrated in the feature space. Furthermore, FR Head could be imposed on different stages of GCNs to build a multi-level refinement for stronger supervision. Extensive experiments are conducted on NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets. Our proposed models obtain competitive results from state-of-the-art methods and can help to discriminate those ambiguous samples. Codes are available at https://github.com/zhysora/FR-Head.

Motivation & Objective

• To address the challenge of ambiguous action recognition in skeleton-based methods, where similar actions (e.g., writing, reading, typing) are hard to distinguish due to missing contextual and object-level cues.
• To improve feature discriminability by refining representations through contrastive learning, focusing on ambiguous samples identified during training.
• To design a lightweight, modular component that can be integrated into various GCN-based backbones for multi-level feature enhancement.
• To achieve better performance on ambiguous action groups without increasing inference cost, enabling plug-and-play deployment.

Proposed method

• The FR-Head decouples raw features into spatial and temporal components to enable focused refinement along each dimension, improving sensitivity to discriminative patterns.
• It identifies confident samples (high prediction confidence) to maintain class prototypes using a contrastive loss that minimizes intra-class distances and maximizes inter-class distances.
• Ambiguous samples are dynamically calibrated by pushing them closer to correct class prototypes or farther from incorrect ones in the feature space.
• The module is applied at multiple stages of GCN backbones, enabling multi-level refinement with joint optimization of classification and contrastive losses.
• The framework uses a momentum-based prototype update strategy to stabilize training and improve generalization.
• The method is compatible with existing GCN models and can be removed at inference, ensuring no runtime overhead.

Experimental results

Research questions

• RQ1 Can contrastive learning be effectively leveraged to improve discriminative representation learning for ambiguous actions in skeleton-based action recognition?
• RQ2 How can spatial and temporal features be decoupled and refined separately to enhance model sensitivity to subtle action differences?
• RQ3 Can a plug-and-play module improve performance on ambiguous action groups without modifying the backbone architecture?
• RQ4 To what extent does multi-level refinement across GCN stages improve recognition accuracy on challenging action categories?
• RQ5 How does the proposed method compare to state-of-the-art models in terms of accuracy and robustness on ambiguous action groups?

Key findings

• The proposed FR-Head achieves a top-1 accuracy of 92.8% on the X-Sub split of NTU RGB+D 120, outperforming the previous SOTA method CTR-GCN by 0.4%.
• On the X-View split of NTU RGB+D, the method achieves 96.8% top-1 accuracy, ranking first among all methods reported.
• On the NW-UCLA dataset, the model reaches 96.8% top-1 accuracy, setting a new SOTA result.
• In group-wise evaluation, the method improves accuracy by up to 1.4 percentage points on ambiguous action groups compared to CTR-GCN.
• t-SNE visualization confirms that the proposed method produces more compact and well-separated feature clusters for ambiguous actions.
• The ablation study validates that both spatial-temporal decoupling and contrastive refinement are essential for performance gains, with each component contributing significantly to discriminability.