[Paper Review] ViT-HGR: Vision Transformer-based Hand Gesture Recognition from High Density Surface EMG Signals

Mansooreh Montazerin, Soheil Zabihi+3 more|arXiv (Cornell University)|Jan 25, 2022

Muscle activation and electromyography studies4 citations

TL;DR

This paper proposes ViT-HGR, a Vision Transformer-based framework for high-density surface EMG (HD-sEMG)-based hand gesture recognition, converting HD-sEMG signals into a patch-based input format compatible with ViT architectures. It achieves 84.62% average test accuracy on 65 gestures with only 78,210 parameters, training from scratch without data augmentation or transfer learning, outperforming LDA in accuracy and training efficiency.

ABSTRACT

Recently, there has been a surge of significant interest on application of Deep Learning (DL) models to autonomously perform hand gesture recognition using surface Electromyogram (sEMG) signals. DL models are, however, mainly designed to be applied on sparse sEMG signals. Furthermore, due to their complex structure, typically, we are faced with memory constraints; require large training times and a large number of training samples, and; there is the need to resort to data augmentation and/or transfer learning. In this paper, for the first time (to the best of our knowledge), we investigate and design a Vision Transformer (ViT) based architecture to perform hand gesture recognition from High Density (HD-sEMG) signals. Intuitively speaking, we capitalize on the recent breakthrough role of the transformer architecture in tackling different complex problems together with its potential for employing more input parallelization via its attention mechanism. The proposed Vision Transformer-based Hand Gesture Recognition (ViT-HGR) framework can overcome the aforementioned training time problems and can accurately classify a large number of hand gestures from scratch without any need for data augmentation and/or transfer learning. The efficiency of the proposed ViT-HGR framework is evaluated using a recently-released HD-sEMG dataset consisting of 65 isometric hand gestures. Our experiments with 64-sample (31.25 ms) window size yield average test accuracy of 84.62 +/- 3.07%, where only 78, 210 number of parameters is utilized. The compact structure of the proposed ViT-based ViT-HGR framework (i.e., having significantly reduced number of trainable parameters) shows great potentials for its practical application for prosthetic control.

Motivation & Objective

• To address the limitations of traditional DL models in handling high-density sEMG signals, particularly memory constraints and long training times.
• To explore the feasibility and effectiveness of Vision Transformers (ViT) in classifying complex, multi-channel HD-sEMG signals for hand gesture recognition.
• To develop a compact, end-to-end deep learning framework that operates directly on raw HD-sEMG data without requiring extensive feature engineering or data augmentation.
• To evaluate the performance of ViT-HGR on a large-scale HD-sEMG dataset with 65 isometric gestures across 19 participants, focusing on accuracy, parameter efficiency, and training speed.

Proposed method

• HD-sEMG signals are segmented into 64-sample (31.25 ms) windows and converted into a patch-based format suitable for Vision Transformer input.
• The proposed ViT-HGR framework employs a standard Vision Transformer architecture with learnable class tokens and positional embeddings, adapted to the spatial-temporal structure of HD-sEMG signals.
• The model is trained end-to-end from scratch using cross-entropy loss, without transfer learning or data augmentation, leveraging the self-attention mechanism to model long-range dependencies across electrodes and time.
• Three variants of the ViT-HGR model are evaluated, differing in depth and width, to study the trade-off between model complexity and performance.
• A 5-fold cross-validation strategy is applied per subject to ensure robust evaluation across the 19 participants in the dataset.
• The framework is compared against a standard LDA classifier using the same 5 key sEMG features (MAV, ZC, WL, RMS, SSC) and AR coefficients for fair benchmarking.

Figure 1: A graphical representation of the Vision Transformer: (a) A cropped portion of the original signal is fed to the ViT and converted to small patches. These patches go through a patch + position embedding layer and a class token is prepended to them. They are then inputted to the transformer

Experimental results

Research questions

• RQ1 Can Vision Transformers be effectively adapted to high-density surface EMG signals for hand gesture recognition without requiring data augmentation or transfer learning?
• RQ2 How does the performance of a ViT-based model compare to conventional machine learning methods like LDA on HD-sEMG data in terms of accuracy and training efficiency?
• RQ3 What is the optimal trade-off between model complexity (number of parameters) and classification accuracy in a ViT-based HD-sEMG framework?
• RQ4 Can a compact ViT architecture achieve high accuracy on a large number of hand gestures (65) using only a small number of parameters?

Key findings

• The ViT-HGR framework achieves an average test accuracy of 84.62% ± 3.07% on 65 isometric hand gestures across 19 participants, using only 78,210 trainable parameters.
• The model outperforms the LDA baseline by approximately 5% in average accuracy and reduces the signal processing and training time by more than 50%, from 9.6 hours to 4.4 hours across all subjects.
• Model II, with fewer parameters (78,210), achieves comparable accuracy (84.62%) to Model I (with ~300,000 parameters), indicating that high performance does not require large model complexity.
• The standard deviation of accuracy across subjects is reduced in the ViT-HGR framework compared to LDA, indicating more consistent performance across individuals.
• The ViT-HGR framework successfully classifies gestures from HD-sEMG signals without any data augmentation or transfer learning, demonstrating its robustness and generalization capability.
• The results suggest that ViTs are well-suited for HD-sEMG due to their ability to model long-range spatial and temporal dependencies efficiently through self-attention, enabling parallel computation and reduced training time.

Figure 2: Accuracy boxplots of 3 different models of the ViT-HGR framework. Each boxplot represents the Interquartile Range for 19 subjects. The accuracy for each subject is the average accuracy after performing 5-fold cross validation.