[Paper Review] Human Pose-based Estimation, Tracking and Action Recognition with Deep Learning: A Survey

Lijuan Zhou, Xiang Meng+4 more|arXiv (Cornell University)|Oct 19, 2023

Human Pose and Action Recognition4 citations

TL;DR

This survey provides a comprehensive, unified review of deep learning-based human pose estimation, tracking, and action recognition, emphasizing their interconnections and integration into end-to-end frameworks. It analyzes state-of-the-art methods across 2D/3D, single/multi-person, image/video settings, and identifies key challenges and future directions in efficiency, zero-shot learning, and multi-modal fusion.

ABSTRACT

Human pose analysis has garnered significant attention within both the research community and practical applications, owing to its expanding array of uses, including gaming, video surveillance, sports performance analysis, and human-computer interactions, among others. The advent of deep learning has significantly improved the accuracy of pose capture, making pose-based applications increasingly practical. This paper presents a comprehensive survey of pose-based applications utilizing deep learning, encompassing pose estimation, pose tracking, and action recognition.Pose estimation involves the determination of human joint positions from images or image sequences. Pose tracking is an emerging research direction aimed at generating consistent human pose trajectories over time. Action recognition, on the other hand, targets the identification of action types using pose estimation or tracking data. These three tasks are intricately interconnected, with the latter often reliant on the former. In this survey, we comprehensively review related works, spanning from single-person pose estimation to multi-person pose estimation, from 2D pose estimation to 3D pose estimation, from single image to video, from mining temporal context gradually to pose tracking, and lastly from tracking to pose-based action recognition. As a survey centered on the application of deep learning to pose analysis, we explicitly discuss both the strengths and limitations of existing techniques. Notably, we emphasize methodologies for integrating these three tasks into a unified framework within video sequences. Additionally, we explore the challenges involved and outline potential directions for future research.

Motivation & Objective

• To provide a comprehensive, unified review of deep learning methods for human pose estimation, tracking, and action recognition, which are often treated separately in prior surveys.
• To analyze the interdependencies among pose estimation, tracking, and action recognition, highlighting how each task supports the others.
• To identify key challenges such as computational complexity, zero-shot learning on skeletons, and multi-modal fusion in action recognition.
• To advocate for the development of end-to-end, unified models that jointly optimize pose estimation, tracking, and action recognition for improved practicality and performance.
• To outline future research directions, including handling occlusion, low-resolution inputs, and balancing accuracy with inference efficiency.

Proposed method

• Systematically categorizes methods across pose estimation (2D/3D, single/multi-person), pose tracking (post-processing vs. integrated, top-down vs. bottom-up), and action recognition (estimated pose vs. skeleton-based).
• Reviews deep learning architectures including CNNs, RNNs, GCNs, and Transformers for action recognition, with emphasis on their application to skeleton sequences.
• Analyzes end-to-end frameworks that jointly perform pose estimation, tracking, and action recognition using RGB video inputs, improving temporal consistency and accuracy.
• Evaluates performance across standard datasets and compares methods using metrics like mAP and accuracy, highlighting trade-offs in complexity and performance.
• Proposes a taxonomy of methods based on input modality (image, video), task type (estimation, tracking, recognition), and architecture (e.g., GCN, Transformer).
• Discusses techniques for efficient attention mechanisms (e.g., token pruning) and lightweight GCNs to reduce computational cost in Transformer- and GCN-based models.

Figure 1: The taxonomy of this survey.

Experimental results

Research questions

• RQ1 How are pose estimation, pose tracking, and action recognition interconnected in deep learning-based human motion analysis?
• RQ2 What are the key limitations of current unified models that jointly perform pose estimation, tracking, and action recognition?
• RQ3 How can computational complexity be reduced in transformer-based models for skeleton-based action recognition?
• RQ4 What are the prospects and challenges of zero-shot learning in skeleton-based action recognition?
• RQ5 How can multi-modal fusion (e.g., RGB, skeleton, text) be effectively and generally implemented for improved action recognition?

Key findings

• The integration of pose estimation, tracking, and action recognition into a unified deep learning framework significantly improves performance and temporal consistency compared to pipeline-based approaches.
• Transformer-based models combined with Graph Neural Networks (GCNs) achieve state-of-the-art accuracy in action recognition, though at the cost of quadratic memory and computation scaling with sequence length.
• Pruning or selecting significant tokens in attention mechanisms can reduce computational cost in transformer-based models without major accuracy loss.
• Zero-shot skeleton-based action recognition remains underexplored, with few methods leveraging textual descriptions to map actions to embeddings for improved generalization.
• Multi-modal fusion using RGB, skeleton, and text data shows promise for distinguishing visually similar actions and enabling zero-shot learning, but lacks generalizable, model-agnostic fusion strategies.
• Current unified models (e.g., UPS) for pose estimation and action recognition underperform compared to separate models, indicating a need for more effective joint optimization frameworks.

Figure 2: The relationship among the three tasks.