[Paper Review] TraHGR: Transformer for Hand Gesture Recognition via ElectroMyography

Soheil Zabihi, Elahe Rahimian+2 more|arXiv (Cornell University)|Mar 28, 2022

Muscle activation and electromyography studies4 citations

TL;DR

This paper proposes TraHGR, a hybrid Transformer-based deep learning framework for hand gesture recognition using sparse multi-channel surface electromyography (sEMG). By combining two parallel pathways—temporal modeling via a Transformer encoder and spatial feature extraction via a CNN—followed by a fusion layer, TraHGR achieves state-of-the-art accuracy of 93.84% on the DB2-D subset (9 gestures), outperforming prior methods by up to 4.30%.

ABSTRACT

Deep learning-based Hand Gesture Recognition (HGR) via surface Electromyogram (sEMG) signals has recently shown significant potential for development of advanced myoelectric-controlled prosthesis. Existing deep learning approaches, typically, include only one model as such can hardly maintain acceptable generalization performance in changing scenarios. In this paper, we aim to address this challenge by capitalizing on the recent advances of hybrid models and transformers. In other words, we propose a hybrid framework based on the transformer architecture, which is a relatively new and revolutionizing deep learning model. The proposed hybrid architecture, referred to as the Transformer for Hand Gesture Recognition (TraHGR), consists of two parallel paths followed by a linear layer that acts as a fusion center to integrate the advantage of each module and provide robustness over different scenarios. We evaluated the proposed architecture TraHGR based on the commonly used second Ninapro dataset, referred to as the DB2. The sEMG signals in the DB2 dataset are measured in the real-life conditions from 40 healthy users, each performing 49 gestures. We have conducted extensive set of experiments to test and validate the proposed TraHGR architecture, and have compared its achievable accuracy with more than five recently proposed HGR classification algorithms over the same dataset. We have also compared the results of the proposed TraHGR architecture with each individual path and demonstrated the distinguishing power of the proposed hybrid architecture. The recognition accuracies of the proposed TraHGR architecture are 86.18%, 88.91%, 81.44%, and 93.84%, which are 2.48%, 5.12%, 8.82%, and 4.30% higher than the state-ofthe-art performance for DB2 (49 gestures), DB2-B (17 gestures), DB2-C (23 gestures), and DB2-D (9 gestures), respectively.

Motivation & Objective

• Address the challenge of low recognition accuracy in sparse multi-channel sEMG-based hand gesture recognition due to limited data and electrode placement sensitivity.
• Overcome the generalization limitations of single-model deep learning approaches in varying real-world scenarios.
• Leverage the attention mechanism of Transformers to model long-range temporal dependencies in sEMG signals while preserving spatial muscle activation patterns.
• Develop a hybrid architecture that fuses temporal and spatial representations for improved robustness and performance across diverse gesture sets.
• Demonstrate superior performance on the benchmark Ninapro DB2 dataset compared to classical ML and existing DNN-based methods.

Proposed method

• Design a dual-path hybrid architecture: one path processes sEMG signals through a Transformer encoder to capture long-range temporal dependencies using multi-head self-attention.
• The second path employs a 1D convolutional neural network (CNN) to extract local spatial patterns from multichannel sEMG signals.
• Integrate the outputs of both pathways via a learnable fusion layer using a linear transformation to combine representations for final classification.
• Train the model using a multi-component loss function that includes cross-entropy loss for the final prediction and auxiliary losses for intermediate supervisions on each pathway.
• Optimize the entire architecture end-to-end using backpropagation with Adam optimizer and learning rate scheduling.
• Apply data augmentation and normalization techniques to improve generalization across subjects and gesture types.

Experimental results

Research questions

• RQ1 Can a hybrid Transformer-CNN architecture outperform single-model deep learning approaches in sEMG-based hand gesture recognition under real-life conditions?
• RQ2 To what extent can the self-attention mechanism in Transformers improve temporal modeling of sparse sEMG signals compared to RNNs or CNNs?
• RQ3 Does fusing temporal (Transformer) and spatial (CNN) representations lead to better generalization across different gesture sets and subjects?
• RQ4 How does the proposed multi-component loss function affect model convergence and recognition accuracy compared to standard cross-entropy loss?
• RQ5 Can the proposed TraHGR framework achieve state-of-the-art performance on the Ninapro DB2 dataset, especially on smaller gesture subsets?

Key findings

• TraHGR achieved a recognition accuracy of 86.18% on the full DB2 dataset (49 gestures), which is 2.48% higher than the state-of-the-art DNN method.
• On the DB2-B subset (17 gestures), TraHGR achieved 88.91% accuracy, a 5.12% improvement over the previous SOTA.
• For the DB2-C subset (23 gestures), the model reached 81.44% accuracy, representing an 8.82% improvement over the prior best method.
• On the DB2-D subset (9 gestures), TraHGR achieved 93.84% accuracy, surpassing the previous SOTA by 4.30%.
• Ablation studies confirmed that the hybrid architecture with joint training and multi-component loss outperformed individual pathways (TNet and FNet), proving the effectiveness of feature fusion.
• The model maintained superior performance even at shorter window sizes—e.g., 84.13% at 100ms, exceeding the 81.1% accuracy of a prior method at 200ms.