[Paper Review] Machine Translation between Spoken Languages and Signed Languages Represented in SignWriting

Zifan Jiang, Amit Moryossef+2 more|Zurich Open Repository and Archive (University of Zurich)|Oct 11, 2022

Hand Gesture Recognition Systems4 citations

TL;DR

This paper presents the first end-to-end machine translation systems between spoken languages and signed languages using SignWriting as an intermediate written representation. By applying neural factored machine translation techniques to parse, factorize, and evaluate SignWriting sequences, the authors achieve over 30 BLEU in bilingual (ASL→English) and over 20 BLEU in multilingual setups, demonstrating that standard NLP techniques can effectively support sign language translation with structured symbolic representations.

ABSTRACT

This paper presents work on novel machine translation (MT) systems between spoken and signed languages, where signed languages are represented in SignWriting, a sign language writing system. Our work seeks to address the lack of out-of-the-box support for signed languages in current MT systems and is based on the SignBank dataset, which contains pairs of spoken language text and SignWriting content. We introduce novel methods to parse, factorize, decode, and evaluate SignWriting, leveraging ideas from neural factored MT. In a bilingual setup--translating from American Sign Language to (American) English--our method achieves over 30 BLEU, while in two multilingual setups--translating in both directions between spoken languages and signed languages--we achieve over 20 BLEU. We find that common MT techniques used to improve spoken language translation similarly affect the performance of sign language translation. These findings validate our use of an intermediate text representation for signed languages to include them in natural language processing research.

Motivation & Objective

• To address the lack of support for signed languages in current machine translation systems by introducing a symbolic intermediate representation using SignWriting.
• To enable end-to-end translation between spoken language text and signed language content by leveraging a sign language writing system.
• To adapt neural factored machine translation techniques to the unique linguistic structure of SignWriting, including symbol parsing and positional factorization.
• To evaluate the performance of sign language translation systems using both standard NLP metrics and novel evaluation strategies tailored to SignWriting’s graphical and spatial features.
• To validate that established low-resource MT techniques are transferable and effective for sign language translation, thereby promoting inclusion of signed languages in NLP research.

Proposed method

• The authors use the SignBank dataset, which provides parallel pairs of spoken language text and corresponding SignWriting sequences, to train translation models.
• They introduce a novel parsing method to segment SignWriting sequences into constituent symbols (graphemes), punctuation, and spatial markers.
• A factorization strategy is applied to separate SignWriting into independent linguistic factors—such as symbols and positional numbers—enabling neural factored MT.
• The models are trained using cross-entropy loss for symbol prediction and mean absolute error (MAE) for positional number prediction, with a focus on preserving spatial relationships.
• The system supports both directionality: from spoken language to SignWriting and vice versa, using shared multilingual architectures.
• Evaluation is conducted using BLEU and chrF2/chrF2++ for symbol-level accuracy, and MAE for positional number prediction, with manual validation of graphical output.

Experimental results

Research questions

• RQ1 Can neural machine translation models effectively learn to translate between spoken language text and SignWriting, a symbolic representation of signed languages?
• RQ2 To what extent do standard low-resource machine translation techniques improve performance in sign language translation when applied to SignWriting?
• RQ3 How can SignWriting’s unique linguistic features—such as spatial positioning and symbolic composition—be effectively modeled in a neural translation framework?
• RQ4 What evaluation metrics are most appropriate for assessing the quality of SignWriting translation outputs, especially given its 2D graphical structure?
• RQ5 Can a multilingual translation setup be effectively trained using limited parallel data, particularly when relying on dictionary-like pairs for non-ASL/English language pairs?

Key findings

• In a bilingual setup translating from American Sign Language to American English, the proposed method achieves a BLEU score of over 30, demonstrating strong performance on a high-resource sign language translation task.
• In two multilingual setups—translating in both directions between spoken languages and signed languages—the system achieves over 20 BLEU, indicating effective generalization across language pairs despite low-resource conditions.
• Common low-resource machine translation techniques, such as data augmentation and multilingual modeling, significantly improve translation performance, confirming their transferability to sign language translation.
• The use of positional numbers as a separate factor in the model improves the accuracy of spatial representation, with MAE used as a key evaluation metric for this component.
• The study reveals limitations in current evaluation practices, particularly the potential flaw in using token-wise MAE for positional numbers when sequence alignment is imperfect.
• The authors advocate for a cascaded evaluation framework that assesses graphical form, symbol composition, positional accuracy, and factorized structure, followed by human evaluation for comprehensive assessment.