[Paper Review] Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation

Necati Cihan Camgöz, Oscar Koller+2 more|arXiv (Cornell University)|Mar 30, 2020

Hand Gesture Recognition Systems4 citations

TL;DR

This paper proposes Sign Language Transformers, a joint end-to-end architecture that simultaneously performs Continuous Sign Language Recognition (CSLR) and Sign Language Translation (SLT) using a transformer-based model with Connectionist Temporal Classification (CTC) loss for gloss-level supervision. The method achieves state-of-the-art results on the PHOENIX14 T dataset, outperforming prior models by more than doubling the BLEU-4 score in some cases (21.80 vs. 9.58), and surpasses even text-to-text translation baselines by directly translating from video representations.

ABSTRACT

Prior work on Sign Language Translation has shown that having a mid-level sign gloss representation (effectively recognizing the individual signs) improves the translation performance drastically. In fact, the current state-of-the-art in translation requires gloss level tokenization in order to work. We introduce a novel transformer based architecture that jointly learns Continuous Sign Language Recognition and Translation while being trainable in an end-to-end manner. This is achieved by using a Connectionist Temporal Classification (CTC) loss to bind the recognition and translation problems into a single unified architecture. This joint approach does not require any ground-truth timing information, simultaneously solving two co-dependant sequence-to-sequence learning problems and leads to significant performance gains. We evaluate the recognition and translation performances of our approaches on the challenging RWTH-PHOENIX-Weather-2014T (PHOENIX14T) dataset. We report state-of-the-art sign language recognition and translation results achieved by our Sign Language Transformers. Our translation networks outperform both sign video to spoken language and gloss to spoken language translation models, in some cases more than doubling the performance (9.58 vs. 21.80 BLEU-4 Score). We also share new baseline translation results using transformer networks for several other text-to-text sign language translation tasks.

Motivation & Objective

• To address the challenge of joint end-to-end learning for sign language recognition and translation without relying on intermediate gloss annotations as bottlenecks.
• To improve translation performance by leveraging recognition supervision through CTC loss within a unified transformer architecture.
• To eliminate the need for explicit gloss tokenization or separate recognition stages, enabling direct video-to-speech translation.
• To demonstrate that joint training enhances both recognition and translation performance simultaneously.
• To establish new SOTA baselines for sign language translation using transformer models across multiple datasets.

Proposed method

• A transformer-based encoder-decoder architecture is used to process sign video sequences and generate spoken language translations.
• Connectionist Temporal Classification (CTC) loss is applied to the encoder to provide gloss-level supervision without requiring ground-truth timing annotations.
• The model jointly optimizes for both sign language recognition and translation in a single end-to-end training process.
• Spatial features are extracted using a 3D CNN backbone, with temporal modeling handled by the transformer encoder and decoder.
• Pretrained sign-specific features (e.g., on sign data) are used to improve representation learning over generic ImageNet-based features.
• The decoder generates spoken language output token-by-token using autoregressive generation, conditioned on the encoded sign features.

Experimental results

Research questions

• RQ1 Can a unified transformer architecture jointly learn sign language recognition and translation in an end-to-end manner without requiring intermediate gloss annotations?
• RQ2 Does injecting CTC loss into the transformer encoder improve both recognition and translation performance?
• RQ3 Can direct video-to-speech translation outperform existing text-to-speech translation baselines that rely on intermediate gloss representations?
• RQ4 How do pretrained sign-specific features compare to generic ImageNet features in improving sign language representation learning?
• RQ5 To what extent does joint learning of recognition and translation enhance performance on both tasks?

Key findings

• The proposed Sign Language Transformers achieve state-of-the-art performance on the PHOENIX14 T dataset, with a BLEU-4 score of 21.80, significantly outperforming prior methods (e.g., 9.58 BLEU-4 in previous work).
• The model surpasses the performance of text-to-text translation baselines, which were previously considered a virtual upper bound, by directly translating from sign video to spoken language.
• Joint training of recognition and translation leads to performance gains on both tasks, demonstrating the co-dependency of the two problems.
• Pretraining on sign-specific data yields better representations than using generic ImageNet-based features, improving both recognition and translation accuracy.
• The model generalizes well to complex linguistic structures, producing grammatically correct and semantically accurate translations despite challenges with named entities and numerical expressions.
• The ablation study confirms that CTC supervision significantly improves recognition quality, which in turn enhances translation performance.