[Paper Review] Code-switching pre-training for neural machine translation

Zhen Yang, Bojie Hu+3 more|arXiv (Cornell University)|Sep 17, 2020

Natural Language Processing Techniques34 references4 citations

TL;DR

This paper proposes Code-Switching Pre-training (CSP), a novel pre-training method for Neural Machine Translation that enhances cross-lingual alignment by randomly replacing source-language words with their target-language translations using unsupervised word embedding mapping. CSP significantly improves both supervised and unsupervised NMT performance and enables better handling of code-switched inputs, achieving +2.3 and +3.0 BLEU points over multi-lingual baselines on in-house test sets.

ABSTRACT

This paper proposes a new pre-training method, called Code-Switching Pre-training (CSP for short) for Neural Machine Translation (NMT). Unlike traditional pre-training method which randomly masks some fragments of the input sentence, the proposed CSP randomly replaces some words in the source sentence with their translation words in the target language. Specifically, we firstly perform lexicon induction with unsupervised word embedding mapping between the source and target languages, and then randomly replace some words in the input sentence with their translation words according to the extracted translation lexicons. CSP adopts the encoder-decoder framework: its encoder takes the code-mixed sentence as input, and its decoder predicts the replaced fragment of the input sentence. In this way, CSP is able to pre-train the NMT model by explicitly making the most of the cross-lingual alignment information extracted from the source and target monolingual corpus. Additionally, we relieve the pretrain-finetune discrepancy caused by the artificial symbols like [mask]. To verify the effectiveness of the proposed method, we conduct extensive experiments on unsupervised and supervised NMT. Experimental results show that CSP achieves significant improvements over baselines without pre-training or with other pre-training methods.

Motivation & Objective

• To address the pretrain-finetune discrepancy caused by artificial tokens like [mask] in existing pre-training methods.
• To leverage cross-lingual alignment signals from monolingual corpora to improve NMT pre-training.
• To develop a production-ready pre-training method that maintains standard NMT model architecture and size.
• To improve NMT performance on code-switching inputs, which are common in real-world multilingual settings.
• To reduce reliance on model fusion or large-scale pre-trained encoders that increase inference cost.

Proposed method

• Perform unsupervised word embedding mapping between source and target languages to induce a bilingual lexicon.
• Randomly replace words in the source sentence with their target-language translations from the induced lexicon, creating a code-mixed input.
• Train an encoder-decoder model where the encoder processes the code-mixed sentence and the decoder predicts the original source words.
• Use a masked language modeling objective over the replaced fragments, enabling the model to learn both sentence-level representations and cross-lingual alignment.
• Pre-train the NMT model using this objective on monolingual source and target corpora without requiring parallel data.
• Fine-tune the pre-trained model on standard NMT tasks with or without back-translation.

Experimental results

Research questions

• RQ1 Can code-switching pre-training improve NMT performance by explicitly leveraging cross-lingual word-pair alignment from monolingual data?
• RQ2 Does CSP reduce the pretrain-finetune discrepancy compared to methods using artificial tokens like [mask]?
• RQ3 Can a pre-trained NMT model handle code-switched inputs more effectively than standard or multi-lingual models?
• RQ4 Which components of the NMT model benefit most from CSP pre-training?
• RQ5 How does CSP compare to existing pre-training methods in both supervised and unsupervised translation settings?

Key findings

• CSP achieves a 28.9 BLEU score on the English-to-German translation task, outperforming the baseline without pre-training (28.4) and other pre-training methods.
• On English-to-French translation, CSP reaches 38.8 BLEU, surpassing the baseline (38.5) and other pre-training baselines.
• On in-house code-switching test sets, CSP improves by +2.3 and +3.0 BLEU points over the multi-lingual baseline on test A and test B, respectively.
• Ablation studies show that the pre-trained encoder and attention module are the most critical components, while the pre-trained decoder has minimal impact due to pretrain-finetune mismatch.
• The method effectively handles code-switched inputs, demonstrating robustness to mixed-language input patterns commonly found in real-world usage.
• CSP maintains model structure and size, making it production-ready without requiring model fusion or additional inference overhead.