[Paper Review] Lexicon-Free Fingerspelling Recognition from Video: Data, Models, and Signer Adaptation

Tae-Hwan Kim, Jonathan Keane+6 more|arXiv (Cornell University)|Sep 26, 2016

Hand Gesture Recognition Systems65 references3 citations

TL;DR

This paper presents a lexicon-free fingerspelling recognition system for American Sign Language using a new multi-signer video dataset and deep learning-based models. It employs segmental conditional random fields with DNN features, achieving 92% accuracy in signer-dependent settings and 83% in multi-signer settings with neural adaptation, advancing robust recognition across signers and recording conditions.

ABSTRACT

We study the problem of recognizing video sequences of fingerspelled letters in American Sign Language (ASL). Fingerspelling comprises a significant but relatively understudied part of ASL. Recognizing fingerspelling is challenging for a number of reasons: It involves quick, small motions that are often highly coarticulated; it exhibits significant variation between signers; and there has been a dearth of continuous fingerspelling data collected. In this work we collect and annotate a new data set of continuous fingerspelling videos, compare several types of recognizers, and explore the problem of signer variation. Our best-performing models are segmental (semi-Markov) conditional random fields using deep neural network-based features. In the signer-dependent setting, our recognizers achieve up to about 92% letter accuracy. The multi-signer setting is much more challenging, but with neural network adaptation we achieve up to 83% letter accuracies in this setting.

Motivation & Objective

• To address the challenge of recognizing continuous fingerspelling in American Sign Language (ASL) across multiple signers with high variability.
• To collect and annotate a new, large-scale multi-signer continuous fingerspelling video dataset to support robust recognition research.
• To develop and evaluate deep learning-based models that generalize across signers despite differences in signing style and video conditions.
• To investigate effective adaptation techniques that improve performance in multi-signer recognition, moving beyond signer-dependent models.

Proposed method

• The authors collected and annotated a new multi-signer continuous fingerspelling video dataset with ground-truth segmentations and peak handshape annotations.
• They used deep neural network (DNN) features extracted from video frames to represent handshape and motion dynamics.
• Segmental (semi-Markov) conditional random fields (CRFs) were employed as the sequence modeling framework to recognize letter sequences.
• Neural network adaptation techniques were applied to improve generalization in the multi-signer setting, reducing performance drop from signer-dependent to multi-signer scenarios.
• The system models fingerspelled letters as discrete handshapes with articulatory features, using peak handshape frames as targets for recognition.
• A phonetic feature representation based on joint angles (MCP, PIP) was defined to describe handshapes consistently across signers.

Experimental results

Research questions

• RQ1 What is the performance of lexicon-free fingerspelling recognition in signer-dependent and multi-signer settings using continuous video data?
• RQ2 How do different sequence modeling architectures (e.g., CRFs vs. HMMs) compare in recognizing continuous fingerspelling sequences?
• RQ3 To what extent can neural network adaptation mitigate performance degradation in multi-signer recognition compared to signer-dependent models?
• RQ4 How effective are DNN-based features in capturing the subtle, coarticulated handshape dynamics in continuous fingerspelling?
• RQ5 What role does high-quality, multi-signer video annotation play in improving recognition accuracy and model generalization?

Key findings

• In the signer-dependent setting, the best-performing model achieved 92% letter accuracy using segmental CRFs with DNN features.
• In the more challenging multi-signer setting, neural network adaptation enabled a recognition accuracy of up to 83%.
• The use of segmental CRFs significantly outperformed simpler models like HMMs, especially in handling coarticulated transitions between letters.
• DNN-based features were critical for capturing fine-grained handshape dynamics and improving robustness across signers.
• The proposed annotation scheme, including peak handshape labeling and phonetic feature definitions, enabled consistent and reliable model training and evaluation.
• The new dataset, while small relative to speech data, is the largest known multi-signer continuous fingerspelling video dataset not restricted to closed vocabularies.