[Paper Review] Unsupervised Translation of German--Lower Sorbian: Exploring Training and Novel Transfer Methods on a Low-Resource Language

Lukas Edman, Ahmet Üstün+2 more|arXiv (Cornell University)|Sep 24, 2021

Natural Language Processing Techniques11 references4 citations

TL;DR

This paper presents a novel unsupervised machine translation system for German–Lower Sorbian, a low-resource language pair, by training on two languages at a time, introducing a vocabulary transfer method from Upper Sorbian to Lower Sorbian, and optimizing the order of online and offline back-translation. The approach achieves state-of-the-art results, ranking first (tied) for DSB→DE and third for DE→DSB, with gains of up to 4.0 BLEU via vocabulary transfer and 2.76 BLEU by using online back-translation first.

ABSTRACT

This paper describes the methods behind the systems submitted by the University of Groningen for the WMT 2021 Unsupervised Machine Translation task for German--Lower Sorbian (DE--DSB): a high-resource language to a low-resource one. Our system uses a transformer encoder-decoder architecture in which we make three changes to the standard training procedure. First, our training focuses on two languages at a time, contrasting with a wealth of research on multilingual systems. Second, we introduce a novel method for initializing the vocabulary of an unseen language, achieving improvements of 3.2 BLEU for DE$ ightarrow$DSB and 4.0 BLEU for DSB$ ightarrow$DE. Lastly, we experiment with the order in which offline and online back-translation are used to train an unsupervised system, finding that using online back-translation first works better for DE$ ightarrow$DSB by 2.76 BLEU. Our submissions ranked first (tied with another team) for DSB$ ightarrow$DE and third for DE$ ightarrow$DSB.

Motivation & Objective

• Address the challenge of unsupervised translation for low-resource languages like Lower Sorbian, which has limited monolingual and no parallel training data.
• Investigate whether fine-tuning on two languages at a time outperforms multilingual pretraining under constrained computational and data resources.
• Develop a method to initialize the word embeddings of an unseen low-resource language (Lower Sorbian) by transferring knowledge from a closely related language (Upper Sorbian).
• Determine the optimal sequence for combining online and offline back-translation to maximize translation quality.
• Establish a benchmark for unsupervised translation in low-resource settings with restricted access to pre-trained multilingual models.

Proposed method

• Train a transformer encoder-decoder model by focusing on two languages at a time—starting with German and Czech, then German and Upper Sorbian, and finally German and Lower Sorbian—based on linguistic similarity.
• Introduce a novel vocabulary transfer method that maps word embeddings from Upper Sorbian to Lower Sorbian using word similarity via Levenshtein distance and top-k candidates.
• Apply a two-stage training process: first pretraining on high-resource language pairs, then fine-tuning on the target low-resource pair with unsupervised training.
• Use both online and offline back-translation, systematically comparing the order: online first versus offline first, to determine the optimal sequence.
• Leverage auxiliary monolingual data from Czech and Upper Sorbian and parallel data from German–Czech to improve initialization and training stability.
• Perform ablation studies to evaluate the impact of vocabulary transfer and back-translation order on BLEU scores.

Experimental results

Research questions

• RQ1 Is it better to pretrain and fine-tune a multilingual model or to focus training on two languages at a time when data and time are limited?
• RQ2 How can we obtain a good initialization for the vocabulary of an unseen low-resource language like Lower Sorbian?
• RQ3 What is the best order for combining online and offline back-translation in unsupervised training?

Key findings

• Using online back-translation before offline back-translation improves DE→DSB translation by 2.76 BLEU compared to the reverse order, indicating that higher-quality synthetic data from online BT enhances subsequent offline training.
• The proposed vocabulary transfer method from Upper Sorbian to Lower Sorbian improves DE→DSB translation by 3.2 BLEU and DSB→DE translation by 4.0 BLEU, with the Levenshtein-based similarity method outperforming a simpler top-1 approach.
• Ablation studies confirm that vocabulary transfer significantly boosts performance, with the Levenshtein method yielding the highest gains, especially in zero-shot DSB→DE translation where it improves from 3.15 to 21.27 BLEU.
• The model achieves a BLEU score of 24.64 on DE→DSB and 27.89 on DSB→DE when using online back-translation followed by offline back-translation, ranking third and first (tied) respectively in the WMT 2021 shared task.
• Even without any fine-tuning on Lower Sorbian, the model with vocabulary transfer achieves a 21.27 BLEU score on DSB→DE translation, demonstrating the critical role of initialization in low-resource settings.
• The results show that unsupervised training benefits substantially from improved vocabulary initialization, with vocabulary transfer closing a 16-18 BLEU gap in zero-shot performance.