[Paper Review] Egyptian Sign Language Recognition Using CNN and LSTM

Ahmed Elhagry, Rawan Gla|arXiv (Cornell University)|Jul 28, 2021

Hand Gesture Recognition Systems8 references4 citations

TL;DR

This paper proposes a video-based Egyptian Sign Language (ESL) recognition system using Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks to address communication gaps between deaf individuals and the hearing population. The CNN extracts spatial features from frames, while the CNN-LSTM architecture captures both spatial and temporal dynamics, achieving 90% accuracy on 9 common ESL words.

ABSTRACT

Sign language is a set of gestures that deaf people use to communicate. Unfortunately, normal people don't understand it, which creates a communication gap that needs to be filled. Because of the variations in (Egyptian Sign Language) ESL from one region to another, ESL provides a challenging research problem. In this work, we are providing applied research with its video-based Egyptian sign language recognition system that serves the local community of deaf people in Egypt, with a moderate and reasonable accuracy. We present a computer vision system with two different neural networks architectures. The first is a Convolutional Neural Network (CNN) for extracting spatial features. The CNN model was retrained on the inception mod. The second architecture is a CNN followed by a Long Short-Term Memory (LSTM) for extracting both spatial and temporal features. The two models achieved an accuracy of 90% and 72%, respectively. We examined the power of these two architectures to distinguish between 9 common words (with similar signs) among some deaf people community in Egypt.

Motivation & Objective

• To bridge the communication gap between deaf individuals using Egyptian Sign Language (ESL) and the hearing population.
• To address regional variations in ESL, which pose challenges for consistent sign language recognition.
• To develop a practical, locally relevant computer vision system for ESL recognition tailored to Egypt’s deaf community.
• To evaluate the performance of two deep learning architectures—CNN and CNN-LSTM—on a localized ESL dataset.
• To achieve moderate to high accuracy in recognizing common ESL words with similar gestures to reduce misclassification.

Proposed method

• Utilized a pre-trained Inception-based CNN model fine-tuned on a custom video dataset of Egyptian Sign Language signs.
• Extracted spatial features from individual video frames using the CNN component to capture hand shape and position.
• Combined the CNN with an LSTM network to model sequential dependencies across frames and capture temporal dynamics of sign movements.
• Trained the CNN-LSTM architecture end-to-end on video sequences to classify 9 common ESL words with similar visual features.
• Employed video data augmentation and transfer learning techniques to improve model generalization on limited local data.
• Evaluated model performance using accuracy metrics on a held-out test set of sign language videos.

Experimental results

Research questions

• RQ1 Can a CNN-based model achieve high accuracy in recognizing Egyptian Sign Language signs using spatial features alone?
• RQ2 To what extent does incorporating temporal modeling via LSTM improve recognition accuracy for dynamic sign language gestures?
• RQ3 How do regional variations in Egyptian Sign Language affect the performance of deep learning-based recognition systems?
• RQ4 What is the performance difference between a standalone CNN and a CNN-LSTM hybrid architecture on a localized ESL dataset?
• RQ5 Can a deep learning system trained on a moderate-sized local dataset achieve practical accuracy for real-world ESL communication applications?

Key findings

• The standalone CNN model achieved a recognition accuracy of 90% on the test set of 9 common Egyptian Sign Language words.
• The CNN-LSTM hybrid model achieved a lower accuracy of 72%, indicating that temporal modeling did not improve performance in this specific setup.
• The results suggest that spatial features alone are highly discriminative for the selected set of similar-sign words in Egyptian Sign Language.
• The high accuracy of the CNN-only model demonstrates the effectiveness of transfer learning with Inception-based architectures on localized sign language recognition.
• The performance gap between the two models indicates that adding LSTM may introduce noise or overfitting when the temporal dynamics are not sufficiently distinct.
• The system shows practical potential for real-world deployment in Egypt’s deaf community due to its high accuracy and localized data focus.