[Paper Review] Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective

Danielle Bragg, Oscar Koller+10 more|arXiv (Cornell University)|Aug 22, 2019

Hand Gesture Recognition SystemsComputer Science95 references86 citations

TL;DR

This paper reports on an interdisciplinary workshop uncovering the current landscape, key challenges, and action items for sign language recognition, generation, and translation across Deaf studies, linguistics, NLP, CV, and HCI.

ABSTRACT

Developing successful sign language recognition, generation, and translation systems requires expertise in a wide range of fields, including computer vision, computer graphics, natural language processing, human-computer interaction, linguistics, and Deaf culture. Despite the need for deep interdisciplinary knowledge, existing research occurs in separate disciplinary silos, and tackles separate portions of the sign language processing pipeline. This leads to three key questions: 1) What does an interdisciplinary view of the current landscape reveal? 2) What are the biggest challenges facing the field? and 3) What are the calls to action for people working in the field? To help answer these questions, we brought together a diverse group of experts for a two-day workshop. This paper presents the results of that interdisciplinary workshop, providing key background that is often overlooked by computer scientists, a review of the state-of-the-art, a set of pressing challenges, and a call to action for the research community.

Motivation & Objective

• Provide an interdisciplinary overview of the current state of sign language processing.
• Identify the major challenges across datasets, recognition, modeling, avatars, and UI/UX.
• Articulate actionable calls to action for cross-disciplinary collaboration and future research.

Proposed method

• Organized a two-day workshop with 39 participants from academia and industry across multiple disciplines.
• Synthesized background knowledge from Deaf culture, linguistics, NLP, CV, graphics, and dataset curation.
• Structured breakout sessions around datasets, recognition, NLP modeling, avatars, and UI/UX to identify state-of-the-art, challenges, and solutions.

Experimental results

Research questions

• RQ1 Q1: What is the current landscape of sign language processing from an interdisciplinary perspective?
• RQ2 Q2: What are the biggest challenges facing the field from an interdisciplinary perspective?
• RQ3 Q3: What calls to action are there for the field that resonate across disciplines?

Key findings

• Datasets remain central and undersized; signer diversity, real-life signing, and native signers are underrepresented.
• Recognition and computer vision face challenges in continuous signing, depictions, annotations, and generalization across signers and contexts.
• Modeling and NLP are hampered by lack of large-scale, reliable annotations and linguistic representations unique to sign languages.
• Avatar and graphics pipelines are not fully automated and must contend with uncanny valley, transition realism, and modulation of signs.
• UI/UX work shows the need for blended text-and-sign interfaces and practical tools for education and accessibility.