[Paper Review] LSA64: An Argentinian Sign Language Dataset

Franco Ronchetti, Facundo Quiroga+3 more|arXiv (Cornell University)|Oct 26, 2023

Hand Gesture Recognition SystemsComputer Science9 references55 citations

TL;DR

This paper introduces LSA64, a research-oriented Argentinian Sign Language dataset with 3200 videos of 64 signs from 10 subjects, plus a preprocessed version and baseline recognition results.

ABSTRACT

Automatic sign language recognition is a research area that encompasses human-computer interaction, computer vision and machine learning. Robust automatic recognition of sign language could assist in the translation process and the integration of hearing-impaired people, as well as the teaching of sign language to the hearing population. Sign languages differ significantly in different countries and even regions, and their syntax and semantics are different as well from those of written languages. While the techniques for automatic sign language recognition are mostly the same for different languages, training a recognition system for a new language requires having an entire dataset for that language. This paper presents a dataset of 64 signs from the Argentinian Sign Language (LSA). The dataset, called LSA64, contains 3200 videos of 64 different LSA signs recorded by 10 subjects, and is a first step towards building a comprehensive research-level dataset of Argentinian signs, specifically tailored to sign language recognition or other machine learning tasks. The subjects that performed the signs wore colored gloves to ease the hand tracking and segmentation steps, allowing experiments on the dataset to focus specifically on the recognition of signs. We also present a pre-processed version of the dataset, from which we computed statistics of movement, position and handshape of the signs.

Motivation & Objective

• Provide a research-grade dataset of Argentinian Sign Language (LSA) to support recognition and machine learning tasks.
• Offer a publicly available resource with both raw and preprocessed data to facilitate reproducibility.
• Characterize the dataset with statistics on handshapes, positions, and trajectories to guide model development.
• Present baseline experiments to establish a reference performance for signer-dependent recognition on LSA64.

Proposed method

• Record 3200 videos of 64 signs performed by 10 subjects using colored gloves to ease hand tracking.
• Provide a preprocessed version with hand/head positions, segmented hand images, and normalized coordinates.
• Describe a baseline sign recognition model that fuses hand position, movement, and handshape information via hand-specific classifiers and product of probabilities.
• Use signer-dependent cross-validation (80-20 split, 30 runs) to report accuracy.
• Compare movement, position, and handshape modalities using Gaussian Mixture Models and Hidden Markov Models in an EM training framework.

Experimental results

Research questions

• RQ1 What is the composition and realism of the LSA64 dataset (sign types, handshapes, movements, subjects)?
• RQ2 Can a baseline signer-dependent model achieve high accuracy on LSA64 using position, movement, and handshape cues?
• RQ3 How do preprocessed features (hand/head positions, segmented hand images) aid recognition compared to raw videos?
• RQ4 What statistics characterize the signs (overlap in movements, initial/final positions, handshapes) to inform model design?
• RQ5 Is the dataset suitable for developing sign language recognition systems for Argentinian Sign Language (LSA)?

Key findings

• LSA64 contains 3200 videos of 64 signs performed by 10 subjects, with both one-handed and two-handed signs.
• Preprocessed data provides hand/head positions and segmented hand images, enabling normalized feature extraction.
• Signer-dependent baseline accuracy achieved 95.95% on the test set (n=30 runs, 80-20 split).
• Baseline uses separate hand classifiers for position, movement, and handshape, with probabilities multiplied across hands for final class likelihoods.
• Movement, position, and handshape cues are modeled with HMM-GMMs and Gaussian distributions in a multi-stream, hand-specific framework.