[Paper Review] GCN-DevLSTM: Path Development for Skeleton-Based Action Recognition

Lei Jiang, Weixin Yang+2 more|arXiv (Cornell University)|Mar 22, 2024

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes GCN-DevLSTM, a novel hybrid model for skeleton-based action recognition that integrates a path development-based LSTM module (G-DevLSTM) into graph convolutional networks to better capture temporal dynamics. By leveraging Lie group-based path development for dimensionality reduction and robustness to irregular sampling, the model achieves state-of-the-art accuracy on NTU60, NTU120, and Chalearn2013 datasets, outperforming existing methods by up to 1.43% on Chalearn2013.

ABSTRACT

Skeleton-based action recognition (SAR) in videos is an important but challenging task in computer vision. The recent state-of-the-art (SOTA) models for SAR are primarily based on graph convolutional neural networks (GCNs), which are powerful in extracting the spatial information of skeleton data. However, it is yet clear that such GCN-based models can effectively capture the temporal dynamics of human action sequences. To this end, we propose the G-Dev layer, which exploits the path development -- a principled and parsimonious representation for sequential data by leveraging the Lie group structure. By integrating the G-Dev layer, the hybrid G-DevLSTM module enhances the traditional LSTM to reduce the time dimension while retaining high-frequency information. It can be conveniently applied to any temporal graph data, complementing existing advanced GCN-based models. Our empirical studies on the NTU60, NTU120 and Chalearn2013 datasets demonstrate that our proposed GCN-DevLSTM network consistently improves the strong GCN baseline models and achieves SOTA results with superior robustness in SAR tasks. The code is available at https://github.com/DeepIntoStreams/GCN-DevLSTM.

Motivation & Objective

• To address the limitation of existing GCN-based models in effectively capturing complex temporal dynamics in skeleton sequences.
• To improve temporal modeling in skeleton-based action recognition beyond standard $K\times1$ convolutional operations.
• To integrate path development from rough path theory into a learnable, efficient, and robust temporal module for sequential graph data.
• To demonstrate the plug-and-play compatibility and performance gains of the proposed G-DevLSTM module across diverse GCN backbones.
• To validate the model’s robustness to frame dropping and irregular sampling in real-world scenarios.

Proposed method

• Proposes a novel G-DevLSTM module that extends path development—originally for vector-valued time series—to graph-structured data with time-varying node features.
• Employs a Lie group structure to represent sequential node features as paths, enabling principled, low-dimensional, and differentiable temporal feature extraction.
• Integrates the path development layer with an LSTM to enhance gradient flow and capture long-range temporal dependencies more effectively than standard convolutional or signature-based methods.
• Uses a topology-non-shared graph (e.g., CTRGCN) where each channel has its own learnable adjacency matrix, preserving spatial structure while enabling flexible temporal modeling.
• Applies the G-DevLSTM module as a plug-in temporal block, replacing standard temporal convolutions in existing GCN frameworks without architectural overhaul.
• Employs a multi-stream architecture where joint features are processed independently through the G-DevLSTM module before fusion, improving discriminative capacity.

Experimental results

Research questions

• RQ1 Can path development from rough path theory be effectively adapted to model temporal dynamics in graph-structured skeleton sequences?
• RQ2 Does the proposed G-DevLSTM module outperform standard $K\times1$ convolutional and signature-based methods in capturing temporal dependencies for action recognition?
• RQ3 How robust is the G-DevLSTM module to missing frames and variable frame rates in real-world skeleton data?
• RQ4 Can the G-DevLSTM module be seamlessly integrated into various GCN backbones to consistently improve performance?
• RQ5 What is the relative contribution of each component in the G-DevLSTM module to overall model accuracy?

Key findings

• On the NTU60 dataset, GCN-DevLSTM achieves 90.8% accuracy on X-sub and 95.7% on X-view, outperforming the previous state-of-the-art CTRGCN by 0.5% and 0.8% respectively.
• On the larger NTU120 dataset, the model achieves 88.7% accuracy on X-sub and 93.1% on X-view, demonstrating strong scalability and performance.
• On the Chalearn2013 gesture dataset, GCN-DevLSTM achieves 95.61% accuracy, surpassing the previous best method by 1.43%.
• The ablation study confirms that removing the path development layer causes the largest performance drop (from 90.8% to 90.5% on X-sub), proving it is the primary contributor to accuracy gains.
• Robustness analysis shows that GCN-DevLSTM maintains superior accuracy across all frame drop rates on NTU60, consistently outperforming both CTRGCN and a GCN-LSTM baseline.
• The model outperforms signature-based methods like Logsig-RNN and Cheng et al. (2023) by 10.4% and 2.75% on NTU120 and Chalearn2013 respectively, validating the advantage of path development over path signature.