[Paper Review] Low-Resource Neural Machine Translation for Southern African Languages

Evander Nyoni, Bruce A. Bassett|arXiv (Cornell University)|Apr 1, 2021

Natural Language Processing Techniques24 references9 citations

TL;DR

This paper proposes multilingual neural machine translation (NMT) to improve low-resource English-to-isiZulu translation using shared representations across related Bantu languages. By jointly training on English-to-isiXhosa and English-to-isiZulu data, the method achieves a 9.9 BLEU point gain over the baseline and sets a new SOTA score of 18.6, significantly outperforming transfer learning and zero-shot learning.

ABSTRACT

Low-resource African languages have not fully benefited from the progress in neural machine translation because of a lack of data. Motivated by this challenge we compare zero-shot learning, transfer learning and multilingual learning on three Bantu languages (Shona, isiXhosa and isiZulu) and English. Our main target is English-to-isiZulu translation for which we have just 30,000 sentence pairs, 28% of the average size of our other corpora. We show the importance of language similarity on the performance of English-to-isiZulu transfer learning based on English-to-isiXhosa and English-to-Shona parent models whose BLEU scores differ by 5.2. We then demonstrate that multilingual learning surpasses both transfer learning and zero-shot learning on our dataset, with BLEU score improvements relative to the baseline English-to-isiZulu model of 9.9, 6.1 and 2.0 respectively. Our best model also improves the previous SOTA BLEU score by more than 10.

Motivation & Objective

• Address the challenge of low-resource neural machine translation (NMT) for Southern African languages, particularly isiZulu, which has only 30,000 parallel sentence pairs.
• Investigate the effectiveness of zero-shot learning, transfer learning, and multilingual learning in low-resource NMT settings for Bantu languages.
• Evaluate how linguistic similarity—especially within the Nguni language sub-class—impacts translation performance.
• Improve the state-of-the-art (SOTA) BLEU score for English-to-isiZulu translation using data-efficient techniques.

Proposed method

• Train a multilingual Transformer-based NMT model jointly on English-to-isiXhosa (E-X) and English-to-isiZulu (E-Z) parallel corpora to share representations across languages.
• Apply transfer learning by fine-tuning a pre-trained English-to-isiXhosa model on the low-resource English-to-isiZulu task.
• Implement zero-shot learning by leveraging the multilingual model to translate English-to-isiZulu without any direct training on that pair.
• Use the Transformer architecture with multi-head self-attention and position-wise feed-forward networks, trained to maximize conditional log-likelihood of target sequences.
• Apply knowledge distillation and shared embedding layers to improve generalization across low-resource language pairs.
• Evaluate performance using BLEU scores on held-out test sets, with statistical significance assessed via standard error estimates.

Experimental results

Research questions

• RQ1 How does multilingual learning compare to transfer learning and zero-shot learning in improving English-to-isiZulu translation quality?
• RQ2 To what extent does linguistic similarity—particularly between Nguni languages like isiXhosa and isiZulu—affect transfer learning performance?
• RQ3 Can zero-shot translation from English to isiZulu surpass transfer learning when using a distant parent model (e.g., English-to-Shona)?
• RQ4 What is the impact of shared multilingual representations on BLEU score improvements for low-resource language pairs?
• RQ5 Does multilingual training significantly outperform fine-tuning from a related language model in low-resource settings?

Key findings

• Multilingual learning achieved a BLEU score of 18.6 ± 1.0 on English-to-isiZulu translation, representing a 9.9 BLEU point improvement over the baseline model.
• The multilingual model surpassed the previous SOTA BLEU score by more than 10 points, setting a new state-of-the-art for English-to-isiZulu translation.
• Transfer learning using the English-to-isiXhosa model yielded a 6.1 BLEU gain over the baseline, while transfer learning from English-to-Shona yielded only 0.9 BLEU, highlighting the role of language similarity.
• The BLEU difference between transfer learning using isiXhosa (6.6) and Shona (0.9) as parent models was 5.2, confirming that linguistic relatedness strongly influences performance.
• Zero-shot learning outperformed transfer learning from Shona by 1.0 BLEU, indicating that distant parent models can be more effective than unrelated ones in zero-shot scenarios.
• The multilingual model achieved a 3.8 BLEU point advantage over the best transfer learning model (E-X parent) and a 9.0 BLEU point advantage over the E-S parent model, demonstrating its superiority in low-resource settings.