[Paper Review] Self-Training for Unsupervised Neural Machine Translation in Unbalanced Training Data Scenarios

Haipeng Sun, Rui Wang+4 more|arXiv (Cornell University)|Apr 9, 2020

Natural Language Processing Techniques19 references4 citations

TL;DR

This paper proposes self-training mechanisms—ST-UT and ST-PT—for unsupervised neural machine translation (UNMT) in unbalanced monolingual data scenarios, where one language has significantly less monolingual data than the other. By leveraging synthetic parallel data generated through back-translation and iteratively refining the model, the proposed methods improve translation performance by 2–4 BLEU points over baseline UNMT, especially benefiting low-resource languages.

ABSTRACT

Unsupervised neural machine translation (UNMT) that relies solely on massive monolingual corpora has achieved remarkable results in several translation tasks. However, in real-world scenarios, massive monolingual corpora do not exist for some extremely low-resource languages such as Estonian, and UNMT systems usually perform poorly when there is not adequate training corpus for one language. In this paper, we first define and analyze the unbalanced training data scenario for UNMT. Based on this scenario, we propose UNMT self-training mechanisms to train a robust UNMT system and improve its performance in this case. Experimental results on several language pairs show that the proposed methods substantially outperform conventional UNMT systems.

Motivation & Objective

• Address the performance degradation of unsupervised neural machine translation (UNMT) when one language has significantly less monolingual data than the other, a scenario termed 'unbalanced training data'.
• Investigate the failure mode in UNMT where the model overfits to the low-resource language while underutilizing the high-resource language's large corpus.
• Develop self-training strategies that effectively leverage the abundant monolingual data in high-resource languages to improve translation quality in low-resource settings.
• Demonstrate that synthetic parallel data generated via back-translation can be effectively integrated into UNMT training to enhance robustness and performance.

Proposed method

• Define the unbalanced training data scenario as a setting where monolingual data sizes for two languages differ significantly, e.g., 50M English vs. 2M French sentences.
• Propose ST-UT (self-training with unsupervised training): use synthetic parallel data generated from the previous model iteration as target-only data in the next training step.
• Propose ST-PT (self-training with pseudo-supervised training): use synthetic parallel data as both source and target in the training objective, enabling more direct integration of pseudo-parallel data.
• Integrate the self-training mechanism into a transformer-based XLM framework with cross-lingual pretraining and denoising auto-encoding.
• Train the model using back-translation loss: maximize the expected log-likelihood of reconstructing source from target and vice versa using pseudo-parallel pairs.
• Use a shared BPE vocabulary of 60,000 subwords and case-sensitive 4-gram BLEU for evaluation across multiple language pairs (En-Fr, En-Ro, En-Et).

Experimental results

Research questions

• RQ1 How does UNMT performance degrade when monolingual data for one language is drastically reduced compared to the other?
• RQ2 Can self-training mechanisms effectively utilize large monolingual corpora in high-resource languages to improve translation in low-resource settings?
• RQ3 Does integrating synthetic parallel data as both source and target (ST-PT) yield better performance than using it only as target (ST-UT)?
• RQ4 To what extent can self-training mitigate overfitting and improve convergence in unbalanced UNMT scenarios?
• RQ5 How does the proposed method compare to standard UNMT baselines in terms of BLEU score across diverse low-resource language pairs?

Key findings

• In unbalanced settings (e.g., 50M En vs. 2M Fr), UNMT performance drops by 4–5 BLEU points compared to balanced 50M/50M data, indicating poor utilization of large monolingual corpora.
• The ST-PT strategy outperforms ST-UT by 1 BLEU point on average, demonstrating that dual-use of synthetic data enhances model learning.
• The proposed self-training mechanisms improve baseline UNMT performance by 2–4 BLEU points across En-Fr, En-Ro, and En-Et language pairs.
• On the En-Fr pair with only 2M French monolingual sentences, ST-PT achieves a BLEU score of 31.84, significantly outperforming the baseline of 30.91.
• Case studies show that ST-PT improves fluency and accuracy: e.g., translating 'tööd ma ei karda' as 'I’m not afraid of work' instead of 'work I’m not afraid of'.
• The model converges better and avoids overfitting in low-resource languages when using self-training, particularly with ST-PT, due to more effective use of synthetic data.