[Paper Review] Integrating Unsupervised Data Generation into Self-Supervised Neural Machine Translation for Low-Resource Languages

Dana Ruiter, Dietrich Klakow+2 more|arXiv (Cornell University)|Jul 19, 2021

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes integrating unsupervised data generation techniques—back-translation and multilingual denoising autoencoding—into self-supervised neural machine translation (SSNMT) to improve low-resource machine translation. By combining online sentence pair extraction with back-translation and multilingual pre-training, the method achieves state-of-the-art results on low-resource and distant language pairs, outperforming both SSNMT and unsupervised MT baselines by up to +4.3 BLEU, even when trained exclusively on monolingual data.

ABSTRACT

For most language combinations, parallel data is either scarce or simply unavailable. To address this, unsupervised machine translation (UMT) exploits large amounts of monolingual data by using synthetic data generation techniques such as back-translation and noising, while self-supervised NMT (SSNMT) identifies parallel sentences in smaller comparable data and trains on them. To date, the inclusion of UMT data generation techniques in SSNMT has not been investigated. We show that including UMT techniques into SSNMT significantly outperforms SSNMT and UMT on all tested language pairs, with improvements of up to +4.3 BLEU, +50.8 BLEU, +51.5 over SSNMT, statistical UMT and hybrid UMT, respectively, on Afrikaans to English. We further show that the combination of multilingual denoising autoencoding, SSNMT with backtranslation and bilingual finetuning enables us to learn machine translation even for distant language pairs for which only small amounts of monolingual data are available, e.g. yielding BLEU scores of 11.6 (English to Swahili).

Motivation & Objective

• Address the challenge of low-resource neural machine translation where parallel data is scarce or nonexistent.
• Investigate whether unsupervised data generation techniques (e.g., back-translation, denoising) can enhance self-supervised NMT (SSNMT) performance.
• Explore the effectiveness of combining multilingual denoising autoencoding, back-translation, and bilingual fine-tuning in low-resource settings.
• Evaluate the method across diverse language pairs, including typologically distant and low-resource languages with minimal monolingual data.
• Demonstrate that SSNMT can achieve strong performance even without parallel data by leveraging unsupervised data augmentation and multilingual pre-training.

Proposed method

• Adapt a self-supervised NMT framework that jointly learns translation and extracts similar sentence pairs from comparable monolingual corpora using a margin-based similarity measure on encoder embeddings.
• Integrate back-translation as an unsupervised data augmentation technique by generating synthetic parallel data through autoencoding and translation loops.
• Apply multilingual denoising autoencoding (MDAE) pre-training on a small set of multilingual monolingual data to improve representation learning across languages.
• Use bilingual fine-tuning to align the model’s output distributions with target language characteristics, improving generalization on low-resource pairs.
• Train the model in an end-to-end, online fashion where sentence pair extraction and translation are iteratively refined using shared encoder-decoder architectures.
• Leverage a shared multilingual encoder with language-specific decoders to enable zero-shot and few-shot transfer to distant language pairs.

Experimental results

Research questions

• RQ1 Can unsupervised data generation techniques such as back-translation and denoising improve the performance of self-supervised NMT in low-resource settings?
• RQ2 How does combining MDAE pre-training, back-translation, and bilingual fine-tuning affect translation quality on low-resource and distant language pairs?
• RQ3 Does the integration of UMT-style data augmentation into SSNMT lead to better generalization than either approach alone?
• RQ4 Can SSNMT with data augmentation achieve competitive performance compared to supervised systems trained on extracted parallel data, even without any parallel monolingual data?
• RQ5 What role does language typology and distance play in the success of the proposed method, and how can multilingual pre-training be optimized for distant language pairs?

Key findings

• The proposed UMT-enhanced SSNMT framework outperforms the SSNMT baseline by up to +4.3 BLEU on Afrikaans-to-English translation, with gains of +50.8 and +51.5 BLEU over statistical UMT and hybrid UMT, respectively.
• On the distant language pair English-to-Swahili, the model achieves a BLEU score of 11.6 using only 66k monolingual sentences, demonstrating effectiveness in extreme low-resource settings.
• The method surpasses a supervised NMT system trained on Laser-extracted parallel sentences for English–Afrikaans and English–Nepali, despite the absence of any parallel data during training.
• For English–Yorùbá, the model achieves a BLEU score of 22.4 with only 501 parallel sentences, outperforming unsupervised and self-supervised baselines.
• The combination of multilingual denoising autoencoding, back-translation, and bilingual fine-tuning enables successful training on language pairs where prior unsupervised or self-supervised methods fail.
• The model’s performance degrades when the source language is not represented in the multilingual pre-training set, highlighting the importance of including related languages in pre-training.