[Paper Review] Skeleton-based Gesture Recognition Using Several Fully Connected Layers with Path Signature Features and Temporal Transformer Module

Chenyang Li, Xin Zhang+3 more|arXiv (Cornell University)|Nov 17, 2018

Human Pose and Action Recognition42 references4 citations

TL;DR

This paper proposes a skeleton-based gesture recognition framework using path signature features (S_PS, T_PS, T_S_PS) and a temporal transformer module (TTM) to model spatial-temporal dynamics. By applying an 'attention on hand' principle for joint selection and a dyadic method for feature extraction, the method achieves state-of-the-art accuracy on ChaLearn 2013, ChaLearn 2016, and MSRC-12 with minimal fully connected layers and high computational efficiency.

ABSTRACT

The skeleton based gesture recognition is gaining more popularity due to its wide possible applications. The key issues are how to extract discriminative features and how to design the classification model. In this paper, we first leverage a robust feature descriptor, path signature (PS), and propose three PS features to explicitly represent the spatial and temporal motion characteristics, i.e., spatial PS (S_PS), temporal PS (T_PS) and temporal spatial PS (T_S_PS). Considering the significance of fine hand movements in the gesture, we propose an "attention on hand" (AOH) principle to define joint pairs for the S_PS and select single joint for the T_PS. In addition, the dyadic method is employed to extract the T_PS and T_S_PS features that encode global and local temporal dynamics in the motion. Secondly, without the recurrent strategy, the classification model still faces challenges on temporal variation among different sequences. We propose a new temporal transformer module (TTM) that can match the sequence key frames by learning the temporal shifting parameter for each input. This is a learning-based module that can be included into standard neural network architecture. Finally, we design a multi-stream fully connected layer based network to treat spatial and temporal features separately and fused them together for the final result. We have tested our method on three benchmark gesture datasets, i.e., ChaLearn 2016, ChaLearn 2013 and MSRC-12. Experimental results demonstrate that we achieve the state-of-the-art performance on skeleton-based gesture recognition with high computational efficiency.

Motivation & Objective

• To address the challenge of discriminative feature extraction in skeleton-based gesture recognition, especially for fine hand movements.
• To design a lightweight, efficient classification model that captures temporal dependencies without recurrent networks.
• To explicitly model both global and local temporal dynamics in gesture sequences using path signature features.
• To improve recognition robustness and efficiency through joint selection strategies and end-to-end trainable temporal alignment.

Proposed method

• Introduces three path signature features: spatial PS (S_PS), temporal PS (T_PS), and temporal-spatial PS (T_S_PS), to encode spatial configurations and temporal dynamics.
• Proposes an 'attention on hand' (AOH) principle to select relevant joint pairs for S_PS and single joints for T_PS, enhancing feature compactness and robustness.
• Employs a dyadic method to extract T_PS and T_S_PS features, enabling encoding of both global and local temporal dynamics with reduced dimensionality.
• Designs a temporal transformer module (TTM) that learns sequence-specific temporal shifting parameters to align key frames, improving temporal modeling in a differentiable, end-to-end manner.
• Constructs a multi-stream fully connected network that processes spatial and temporal features separately before early fusion, minimizing model complexity.
• Integrates the TTM and path signature features into a standard neural network architecture, enabling plug-and-play deployment.

Experimental results

Research questions

• RQ1 Can path signature features effectively capture both spatial and temporal dynamics in skeleton-based gesture sequences?
• RQ2 How does the 'attention on hand' principle improve feature compactness and recognition accuracy for fine hand gestures?
• RQ3 Can a learnable temporal transformation module (TTM) outperform fixed or recurrent temporal modeling in gesture recognition?
• RQ4 To what extent can a simple fully connected network with multi-stream feature processing achieve state-of-the-art performance?

Key findings

• The proposed method achieves 92.08% accuracy on ChaLearn 2013, surpassing prior skeleton-based methods and approaching video-based models.
• On ChaLearn 2016, the model reaches 39.95% accuracy, outperforming existing skeleton-based methods by ~4.5 percentage points.
• On MSRC-12, the method achieves 99.01% accuracy, setting a new state-of-the-art result for skeleton-based gesture recognition.
• The model requires only 2.69 million multiply-adds (FLOPs), significantly lower than RNN and CNN baselines, demonstrating high computational efficiency.
• Ablation studies confirm that both the AOH principle and TTM module contribute significantly to performance gains.
• The method maintains high performance even with limited model complexity, proving the effectiveness of explicit feature engineering and lightweight architecture.