[Paper Review] Universal Conditional Masked Language Pre-training for Neural Machine Translation

Pengfei Li, Liangyou Li+3 more|arXiv (Cornell University)|Mar 17, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes CeMAT, a multilingual sequence-to-sequence model pre-trained with a bidirectional decoder using conditional masked language modeling on large-scale monolingual and bilingual corpora. It introduces two novel masking strategies—aligned code-switching & masking and dynamic dual-masking—to enhance cross-lingual representation learning, achieving state-of-the-art gains of up to +14.4 BLEU on low-resource and +7.9 BLEU on average for autoregressive NMT, and up to +5.3 BLEU for non-autoregressive NMT.

ABSTRACT

Pre-trained sequence-to-sequence models have significantly improved Neural Machine Translation (NMT). Different from prior works where pre-trained models usually adopt an unidirectional decoder, this paper demonstrates that pre-training a sequence-to-sequence model but with a bidirectional decoder can produce notable performance gains for both Autoregressive and Non-autoregressive NMT. Specifically, we propose CeMAT, a conditional masked language model pre-trained on large-scale bilingual and monolingual corpora in many languages. We also introduce two simple but effective methods to enhance the CeMAT, aligned code-switching & masking and dynamic dual-masking. We conduct extensive experiments and show that our CeMAT can achieve significant performance improvement for all scenarios from low- to extremely high-resource languages, i.e., up to +14.4 BLEU on low resource and +7.9 BLEU improvements on average for Autoregressive NMT. For Non-autoregressive NMT, we demonstrate it can also produce consistent performance gains, i.e., up to +5.3 BLEU. To the best of our knowledge, this is the first work to pre-train a unified model for fine-tuning on both NMT tasks. Code, data, and pre-trained models are available at https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/CeMAT.

Motivation & Objective

• To improve neural machine translation (NMT) performance by pre-training a unified sequence-to-sequence model with a bidirectional decoder, rather than relying on unidirectional decoders.
• To address the performance gap in non-autoregressive NMT (NAT) by providing a pre-trained initialization that better captures representation capabilities.
• To enhance cross-lingual and intra-lingual representation learning through novel masking strategies that align semantics across languages.
• To unify pre-training for both autoregressive and non-autoregressive NMT tasks, eliminating the need for separate model initialization.
• To investigate whether bidirectional decoders, pre-trained with conditional masking, can outperform standard unidirectional decoders in both generation and representation learning.

Proposed method

• Proposes CeMAT, a multilingual sequence-to-sequence model with a bidirectional encoder, a bidirectional decoder, and a cross-attention module to connect them.
• Pre-trains the model using masked language modeling (MLM) on monolingual data and conditional masked language modeling (CMLM) on bilingual data.
• Introduces aligned code-switching & masking: aligns source words with their target translations, then substitutes source words with semantically similar words in other languages while masking the corresponding target words.
• Applies dynamic dual-masking to mask remaining words in both source and target languages using a dynamic masking strategy that adapts during training.
• Uses large-scale monolingual and bilingual corpora across many languages to pre-train the model in a unified framework.
• Fine-tunes the pre-trained CeMAT model directly on both autoregressive and non-autoregressive NMT tasks without reinitialization.

Experimental results

Research questions

• RQ1 Can a bidirectional decoder in a pre-trained sequence-to-sequence model outperform unidirectional decoders in both autoregressive and non-autoregressive NMT?
• RQ2 Does pre-training a unified model with conditional masking improve cross-lingual representation learning and generalization across low- and high-resource language pairs?
• RQ3 Can aligned code-switching & masking effectively align semantic representations across languages during pre-training?
• RQ4 Does dynamic dual-masking enhance the decoder's ability to attend to relevant source-side context and improve generation quality?
• RQ5 Can a single pre-trained model serve as an effective initialization for both autoregressive and non-autoregressive NMT without task-specific reinitialization?

Key findings

• CeMAT achieves a +14.4 BLEU point improvement over baselines on low-resource NMT settings (less than 1M bitext pairs).
• For autoregressive NMT, CeMAT improves performance by up to +7.9 BLEU on average across multiple language pairs, even in high-resource settings (>25M bitext pairs).
• For non-autoregressive NMT, CeMAT achieves consistent gains of up to +5.3 BLEU, demonstrating its effectiveness in parallel decoding scenarios.
• The model shows faster and more stable convergence during iterative refinement in NAT, requiring only 3–6 iterations to reach peak performance.
• Back-translation further improves performance by +2.1 BLEU on the WMT16 Romanian-to-English task, indicating compatibility with data augmentation techniques.
• CeMAT is the first unified pre-trained model suitable for both autoregressive and non-autoregressive NMT, eliminating the need for separate initialization strategies.