[Paper Review] Tencent's Multilingual Machine Translation System for WMT22 Large-Scale African Languages

Wenxiang Jiao, Zhaopeng Tu+4 more|arXiv (Cornell University)|Oct 18, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper presents Tencent's multilingual neural machine translation system for the WMT22 shared task on large-scale African languages, addressing data scarcity, data imbalance, and the curse of multilinguality through data augmentation, distributionally robust optimization (DRO), and language family grouping. The system achieved first place on the blind test set with a BLEU score of 17.95, significantly outperforming the baseline model.

ABSTRACT

This paper describes Tencent's multilingual machine translation systems for the WMT22 shared task on Large-Scale Machine Translation Evaluation for African Languages. We participated in the $\mathbf{constrained}$ translation track in which only the data and pretrained models provided by the organizer are allowed. The task is challenging due to three problems, including the absence of training data for some to-be-evaluated language pairs, the uneven optimization of language pairs caused by data imbalance, and the curse of multilinguality. To address these problems, we adopt data augmentation, distributionally robust optimization, and language family grouping, respectively, to develop our multilingual neural machine translation (MNMT) models. Our submissions won the $\mathbf{1st\ place}$ on the blind test sets in terms of the automatic evaluation metrics. Codes, models, and detailed competition results are available at https://github.com/wxjiao/WMT2022-Large-Scale-African.

Motivation & Objective

• To address the lack of parallel training data for certain African language pairs in the WMT22 large-scale multilingual translation task.
• To mitigate performance degradation caused by data imbalance across diverse African language pairs.
• To alleviate the curse of multilinguality in multilingual neural machine translation (MNMT) models handling hundreds of language pairs.
• To develop a robust, scalable MNMT system under constrained training conditions using only official data and pretrained models.

Proposed method

• Applied back-translation and self-training with special tags to generate synthetic parallel data for low-resource language pairs without parallel monolingual or bitext data.
• Employed distributionally robust optimization (DRO) to balance model optimization across all translation directions, especially improving performance on dominant languages like English and French.
• Implemented language family grouping (LFG) by clustering languages into families (e.g., Bantu, Nilo-Saharan) and training separate models per group to reduce interference between unrelated language pairs.
• Used a two-stage training pipeline: first pretraining on large-scale data (Large-234), then fine-tuning on cleaner, smaller datasets (Base-146 and Eval-106) to avoid error propagation from synthetic data.
• Integrated target language tags at the encoder to enhance zero-shot transfer, though this did not yield significant gains in the multi-central-language setting.
• Conducted ablation studies to evaluate the effectiveness of each component, including continuous training, fine-tuning, and model scaling.

Experimental results

Research questions

• RQ1 Can data augmentation techniques effectively improve translation performance for low-resource African language pairs with no available parallel data?
• RQ2 To what extent can distributionally robust optimization (DRO) balance model performance across high- and low-resource language pairs in a multilingual setting?
• RQ3 How effective is language family grouping in reducing interference and improving translation quality in multilingual NMT models with hundreds of language pairs?
• RQ4 Does increasing model capacity or applying target language tagging improve performance in a multi-central-language MNMT setup?

Key findings

• The proposed system achieved a BLEU score of 17.95 on the blind test set, surpassing the baseline model (15.50 BLEU) by 2.45 points.
• Data augmentation via back-translation and self-training improved performance when used to pretrain models, but direct training on synthetic data harmed performance due to error propagation.
• Distributionally robust optimization (DRO) significantly improved performance across all language pairs, especially benefiting dominant languages like English and French.
• Language family grouping (LFG) yielded the largest gains on one-to-many translation directions, demonstrating its effectiveness in mitigating the curse of multilinguality.
• Fine-tuning on smaller, cleaner datasets (Eval-106) outperformed direct fine-tuning on larger, noisier datasets (Large-234), indicating that data quality outweighs quantity in multilingual training.
• The model with language family grouping and DRO achieved the highest performance (17.95 BLEU), securing first place in the official WMT22 constrained translation track.