[Paper Review] A Survey on 3D Skeleton-Based Action Recognition Using Learning Method

Bin Ren, Mengyuan Liu+2 more|arXiv (Cornell University)|Feb 14, 2020

Human Pose and Action Recognition75 references86 citations

TL;DR

This survey comprehensively reviews deep learning approaches for 3D skeleton-based action recognition, covering RNNs, CNNs, GCNs, and Transformers, and compares state-of-the-art methods on NTU-RGB+D and NTU-RGB+D 120 datasets.

ABSTRACT

3D skeleton-based action recognition (3D SAR) has gained significant attention within the computer vision community, owing to the inherent advantages offered by skeleton data. As a result, a plethora of impressive works, including those based on conventional handcrafted features and learned feature extraction methods, have been conducted over the years. However, prior surveys on action recognition have primarily focused on video or RGB data-dominated approaches, with limited coverage of reviews related to skeleton data. Furthermore, despite the extensive application of deep learning methods in this field, there has been a notable absence of research that provides an introductory or comprehensive review from the perspective of deep learning architectures. To address these limitations, this survey first underscores the importance of action recognition and emphasizes the significance of 3D skeleton data as a valuable modality. Subsequently, we provide a comprehensive introduction to mainstream action recognition techniques based on four fundamental deep architectures, i.e., Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), Graph Convolutional Network (GCN), and Transformers. All methods with the corresponding architectures are then presented in a data-driven manner with detailed discussion. Finally, we offer insights into the current largest 3D skeleton dataset, NTU-RGB+D, and its new edition, NTU-RGB+D 120, along with an overview of several top-performing algorithms on these datasets. To the best of our knowledge, this research represents the first comprehensive discussion of deep learning-based action recognition using 3D skeleton data.

Motivation & Objective

• Motivate the use of 3D skeleton data as a robust modality for action recognition.
• Systematically summarize deep learning architectures used for 3D SAR (RNNs, CNNs, GCNs, Transformers).
• Analyze data representations, spatial-temporal modeling, and co-occurrence features in skeleton-based methods.
• Provide benchmarks and insights on NTU-RGB+D and NTU-RGB+D 120 to guide future research.

Proposed method

• Introduce four fundamental DL architectures (RNNs, CNNs, GCNs, Transformers) and compare their properties in 3D SAR.
• Discuss data representations and preprocessing strategies for skeleton data (joint/bone graphs, skeleton images, co-occurrence features).
• Survey representative methods within each architecture, focusing on spatial-temporal modeling and attention mechanisms.
• Highlight graph-structured approaches (ST-GCN, 2s-AGCN, MS-G3D, etc.) and transformer-based variants (self-attention, decoupled attention) as core techniques.
• Present a data-driven analysis of datasets and performance trends on NTU-RGB+D and NTU-RGB+D 120.

Experimental results

Research questions

• RQ1 What are the main deep learning architectures used for 3D skeleton-based action recognition and how do they compare?
• RQ2 How do RNNs, CNNs, GCNs, and Transformers handle spatial-temporal modeling and skeleton data representations?
• RQ3 What are the current top-performing methods on NTU-RGB+D and NTU-RGB+D 120, and what architectures do they employ?
• RQ4 What future directions and challenges remain for 3D SAR with skeleton data?

Key findings

• GCN-based methods generally achieve leading results on NTU-RGB+D and NTU-RGB+D 120 among skeleton-based approaches.
• Transformer-based methods show strong potential and are increasingly combined with GCNs or CNNs in hybrid models.
• Recent datasets (NTU-RGB+D 120) present increased difficulty, indicating room for further advancement across architectures.
• Representations that capture joint–bone structure and spatial-temporal graphs, along with adaptive topologies, contribute to performance gains.
• Datasets and evaluation protocols (Cross-Subject, Cross-View, Cross-Setup) are crucial for fair comparisons of 3D SAR models.