[Paper Review] Clean Text and Full-Body Transformer: Microsoft's Submission to the WMT22 Shared Task on Sign Language Translation

Subhadeep Dey, Abhilash Pal+2 more|arXiv (Cornell University)|Oct 24, 2022

Hand Gesture Recognition Systems4 citations

TL;DR

This paper presents Microsoft's submission to the WMT22 shared task on sign language translation, leveraging full-body video features from a pre-trained I3D model and a cleaned text pipeline to achieve the highest BLEU score (0.6) and first-place human evaluation on the test set for Swiss German sign language to spoken language translation. The system further improves to 1.08 BLEU on the dev set using lip-reading features and a lexical dataset, demonstrating the effectiveness of multimodal inputs and data curation in low-resource sign language translation.

ABSTRACT

This paper describes Microsoft's submission to the first shared task on sign language translation at WMT 2022, a public competition tackling sign language to spoken language translation for Swiss German sign language. The task is very challenging due to data scarcity and an unprecedented vocabulary size of more than 20k words on the target side. Moreover, the data is taken from real broadcast news, includes native signing and covers scenarios of long videos. Motivated by recent advances in action recognition, we incorporate full body information by extracting features from a pre-trained I3D model and applying a standard transformer network. The accuracy of the system is further improved by applying careful data cleaning on the target text. We obtain BLEU scores of 0.6 and 0.78 on the test and dev set respectively, which is the best score among the participants of the shared task. Also in the human evaluation the submission reaches the first place. The BLEU score is further improved to 1.08 on the dev set by applying features extracted from a lip reading model.

Motivation & Objective

• To address the challenge of sign language to spoken language translation in a real-world, low-resource setting with high vocabulary diversity.
• To improve translation performance by incorporating full-body video representations using a pre-trained I3D model.
• To enhance model generalization and accuracy through systematic text cleaning and integration of a lexical dataset.
• To evaluate the impact of multimodal inputs—specifically full-body and lip-reading features—on translation quality in sign language translation.

Proposed method

• Extracted video embeddings using a pre-trained I3D model to encode full-body sign language motion from video frames.
• Applied a standard Transformer decoder architecture to generate spoken language translations from the I3D-embedded sequences.
• Performed extensive text preprocessing to reduce vocabulary size and singleton words, improving model generalization and training efficiency.
• Integrated a pre-trained AV-HuBERT lip-reading model to extract complementary visual features from the mouth region, enhancing recognition of low-frequency and function words.
• Combined full-body and lip-reading features in a joint input representation to improve sequence modeling and translation accuracy.
• Incorporated a public sign language lexicon (SignSuisse) to boost representation of high-frequency, function words that were poorly predicted by the base model.

Experimental results

Research questions

• RQ1 Can full-body video features from a pre-trained I3D model significantly improve sign language translation performance in a low-resource setting?
• RQ2 To what extent does text cleaning reduce vocabulary size and improve BLEU scores in sign language translation systems?
• RQ3 How effective are lip-reading features from AV-HuBERT in improving recognition of rare or function words in sign language translation?
• RQ4 Does integrating a lexical dataset enhance the model's ability to predict high-frequency, low-ambiguity signs that are often missed by end-to-end models?
• RQ5 How do the performance metrics on the WMT22 shared task compare to established benchmarks like PHOENIX-2014T, and what does this imply about task difficulty?

Key findings

• The system achieved a BLEU score of 0.6 on the official WMT2022 test set, the highest among all 7 participants.
• Human evaluation ranked the system first, confirming its strong performance in naturalness and fluency of generated translations.
• The addition of lip-reading features from AV-HuBERT improved the reduced BLEU score from 0.78 (I3D only) to 1.08 on the dev set.
• The integration of a lexical dataset further enhanced performance, particularly for function words like 'auf wiedersehen', which were previously poorly predicted.
• The model trained with WMT data performed significantly worse than state-of-the-art models on PHOENIX-2014T, indicating that the WMT22 task is substantially more challenging due to data quality and domain constraints.
• The presence of sentence-final full stops in PHOENIX-2014T data led to a 1% relative BLEU improvement, highlighting the sensitivity of BLEU to punctuation and formatting.