[Paper Review] Quantitative Survey of the State of the Art in Sign Language Recognition

Oscar Koller|arXiv (Cornell University)|Aug 22, 2020

Hand Gesture Recognition Systems278 references58 citations

TL;DR

This paper provides a quantitative survey of sign language recognition approaches, highlighting modality usage across results, with Table S6 detailing 26 modality combinations and their relative frequencies by vocabulary size.

ABSTRACT

This work presents a meta study covering around 300 published sign language recognition papers with over 400 experimental results. It includes most papers between the start of the field in 1983 and 2020. Additionally, it covers a fine-grained analysis on over 25 studies that have compared their recognition approaches on RWTH-PHOENIX-Weather 2014, the standard benchmark task of the field. Research in the domain of sign language recognition has progressed significantly in the last decade, reaching a point where the task attracts much more attention than ever before. This study compiles the state of the art in a concise way to help advance the field and reveal open questions. Moreover, all of this meta study's source data is made public, easing future work with it and further expansion. The analyzed papers have been manually labeled with a set of categories. The data reveals many insights, such as, among others, shifts in the field from intrusive to non-intrusive capturing, from local to global features and the lack of non-manual parameters included in medium and larger vocabulary recognition systems. Surprisingly, RWTH-PHOENIX-Weather with a vocabulary of 1080 signs represents the only resource for large vocabulary continuous sign language recognition benchmarking world wide.

Motivation & Objective

• Assess the distribution of sign language recognition modalities used in published results
• Quantify how modality choices vary with vocabulary size (e.g., number of signs)
• Identify which manual and non-manual parameter combinations are most prevalent in the literature
• Provide a concise reference for researchers choosing modalities for SLR experiments

Proposed method

• Collect published sign language recognition results with defined vocabulary ranges
• Classify each result by the modality combination used (Loc, Mov, Shape, Orient, Joints, Fullframe, Depth, Motion)
• Compute relative frequencies of each modality combination within each vocabulary range
• Present summarized statistics (e.g., Table S6) showing prevalence of modalities
• Note any truncations or data quality issues in shared supplementary material

Experimental results

Research questions

• RQ1 What modality combinations are most commonly used in sign language recognition studies?
• RQ2 How does modality usage change as the modeled vocabulary size increases?
• RQ3 What is the relative prevalence of fullframe versus targeted hand/shape/orientation modalities across vocabulary ranges?
• RQ4 Are there notable trends in manual vs non-manual parameter usage in the literature?

Key findings

• Table S6 reports the 26 most frequently used modality combinations and their relative frequencies within vocabulary ranges.
• The results are presented as percentages relative to all results in the same vocabulary range.
• An illustrative example: 39% of results with a modeled vocabulary above 1000 signs rely fully on the fullframe modality, while 7% rely on the hand shape modality.