[Paper Review] PSUMNet: Unified Modality Part Streams are All You Need for Efficient Pose-based Action Recognition

Neel Trivedi, Ravi Kiran Sarvadevabhatla|arXiv (Cornell University)|Aug 11, 2022

Human Pose and Action Recognition4 citations

TL;DR

PSUMNet introduces a unified modality part stream architecture for efficient pose-based action recognition, replacing conventional independent modality streams with part-based processing that captures localized dynamics (e.g., hands for 'Thumbs up', legs for 'Kicking'). It achieves state-of-the-art accuracy on NTU RGB+D 60/120 and NTU-X datasets with 100%-400% fewer parameters than prior methods, enabling efficient deployment on edge devices.

ABSTRACT

Pose-based action recognition is predominantly tackled by approaches which treat the input skeleton in a monolithic fashion, i.e. joints in the pose tree are processed as a whole. However, such approaches ignore the fact that action categories are often characterized by localized action dynamics involving only small subsets of part joint groups involving hands (e.g. `Thumbs up') or legs (e.g. `Kicking'). Although part-grouping based approaches exist, each part group is not considered within the global pose frame, causing such methods to fall short. Further, conventional approaches employ independent modality streams (e.g. joint, bone, joint velocity, bone velocity) and train their network multiple times on these streams, which massively increases the number of training parameters. To address these issues, we introduce PSUMNet, a novel approach for scalable and efficient pose-based action recognition. At the representation level, we propose a global frame based part stream approach as opposed to conventional modality based streams. Within each part stream, the associated data from multiple modalities is unified and consumed by the processing pipeline. Experimentally, PSUMNet achieves state of the art performance on the widely used NTURGB+D 60/120 dataset and dense joint skeleton dataset NTU 60-X/120-X. PSUMNet is highly efficient and outperforms competing methods which use 100%-400% more parameters. PSUMNet also generalizes to the SHREC hand gesture dataset with competitive performance. Overall, PSUMNet's scalability, performance and efficiency makes it an attractive choice for action recognition and for deployment on compute-restricted embedded and edge devices. Code and pretrained models can be accessed at https://github.com/skelemoa/psumnet

Motivation & Objective

• To address the inefficiency and monolithic processing of existing pose-based action recognition methods that treat skeletons as whole entities.
• To overcome the limitations of independent modality streams, which increase parameters and ignore inter-modality correlations.
• To improve performance on actions dominated by specific joint groups (e.g., hands, legs) by introducing part-based stream processing.
• To enable scalability across diverse skeleton configurations, including dense (NTU-X) and sparse (SHREC) datasets.
• To develop a lightweight, efficient architecture suitable for real-world deployment on compute-constrained edge and embedded devices.

Proposed method

• Proposes a unified modality processing pipeline where multiple modalities (joint, bone, joint velocity, bone velocity) are fused within each part stream instead of using separate streams.
• Introduces part-based streams (body, hands, legs) that process sub-skeletons of the full pose in a global frame of reference, enabling localized and context-aware representations.
• Uses overlapping joint groups across streams to preserve global motion propagation while enabling specialized processing for localized actions.
• Employs a shared backbone network per part stream with modality fusion at the input level, reducing parameter count compared to independent stream training.
• Applies late fusion of part stream predictions to combine specialized representations without increasing model complexity per stream.
• Designs the architecture to be scalable across datasets with varying joint density, including full-body (NTURGB+D), dense (NTU-X), and sparse (SHREC) configurations.

Experimental results

Research questions

• RQ1 Can a unified modality processing strategy reduce parameter count while maintaining or improving performance in pose-based action recognition?
• RQ2 Does part-based stream processing outperform monolithic or disjoint part processing in capturing localized action dynamics?
• RQ3 How does the proposed architecture generalize across diverse skeleton datasets with varying joint density and configuration?
• RQ4 To what extent do inter-modality correlations contribute to performance when processed jointly within each part stream?
• RQ5 Can the proposed method achieve state-of-the-art accuracy with significantly fewer parameters than existing multi-stream approaches?

Key findings

• PSUMNet achieves 92.9% accuracy on NTU60 cross-subject split and 89.4% on NTU120, outperforming state-of-the-art methods with 100%-400% fewer parameters.
• The body stream alone achieves 91.9% accuracy on NTU60, while the combined body+hands stream reaches 92.4%, demonstrating the benefit of specialized part streams.
• The legs stream achieves 60.4% accuracy on NTU60, indicating that actions dominated by leg movements are less frequent but still effectively captured.
• PSUMNet outperforms CTR-GCN in early action recognition, maintaining higher accuracy even when only 30%-50% of the sequence is observed.
• Ablation studies confirm that joint and bone modalities contribute most to performance, and using all four modalities yields the best result.
• The unified modality approach reduces parameter count significantly compared to independent stream training, with PSUMNet using only 2.8M parameters despite fusing four modalities.