[Paper Review] Target Conditioned Sampling: Optimizing Data Selection for Multilingual Neural Machine Translation

Xinyi Wang, Graham Neubig|arXiv (Cornell University)|May 20, 2019

Natural Language Processing Techniques21 references4 citations

TL;DR

This paper proposes Target Conditioned Sampling (TCS), a data selection method that optimizes multilingual neural machine translation by sampling source sentences conditionally based on target sentence relevance, minimizing training loss for low-resource languages. TCS achieves up to 2 BLEU gains over strong baselines with minimal training overhead, demonstrating significant improvements in low-resource translation performance while maintaining efficiency.

ABSTRACT

To improve low-resource Neural Machine Translation (NMT) with multilingual corpora, training on the most related high-resource language only is often more effective than using all data available (Neubig and Hu, 2018). However, it is possible that an intelligent data selection strategy can further improve low-resource NMT with data from other auxiliary languages. In this paper, we seek to construct a sampling distribution over all multilingual data, so that it minimizes the training loss of the low-resource language. Based on this formulation, we propose an efficient algorithm, Target Conditioned Sampling (TCS), which first samples a target sentence, and then conditionally samples its source sentence. Experiments show that TCS brings significant gains of up to 2 BLEU on three of four languages we test, with minimal training overhead.

Motivation & Objective

• To address the challenge of improving low-resource neural machine translation (NMT) using multilingual parallel corpora.
• To design a data selection strategy that intelligently selects relevant multilingual training data to improve NMT performance and training speed.
• To minimize the training loss of low-resource language models by constructing an optimal sampling distribution over multilingual data.
• To develop an efficient, target-invariant sampling framework that leverages conditional source invariance for improved generalization.
• To evaluate the effectiveness of TCS across multiple low-resource languages using diverse similarity metrics and model architectures.

Proposed method

• TCS constructs a sampling distribution Q(X,Y) that approximates the true source-target distribution P_s(X,Y) while ensuring target invariance Q(Y) ≈ P_s(Y).
• The method first samples a target sentence y from Q(Y), then conditionally samples a source sentence x from Q(X|y), where Q(X|y) is derived from similarity between source and target languages.
• TCS uses language-level or sentence-level similarity metrics (e.g., LM, vocabulary overlap) to estimate Q(X|y), with options for diversity-aware sampling (TCS-S) or document-level similarity (TCS-D).
• The framework is designed to be efficient, requiring no additional pretraining, and integrates seamlessly into standard NMT training pipelines.
• TCS is evaluated with both standard NMT models and soft decoupled encoding (SDE) models to assess generalization across architectures.
• The method is grounded in a mathematical formulation that ensures the expected loss over Q(X,Y) correlates with the true loss over P_s(X,Y), enabling effective optimization.

Experimental results

Research questions

• RQ1 Can a data selection strategy that samples multilingual data based on target relevance improve low-resource neural machine translation beyond using only high-resource language data?
• RQ2 How can we construct a sampling distribution Q(X,Y) that minimizes the training loss for low-resource language models while maintaining target invariance?
• RQ3 What are the most effective similarity metrics for estimating Q(X|y) in low-resource settings, and how do they impact performance?
• RQ4 Does TCS maintain training efficiency while improving translation quality, especially when compared to full-data training?
• RQ5 How does TCS generalize across different multilingual NMT architectures, such as those using soft decoupled encoding (SDE)?

Key findings

• TCS achieves up to 2 BLEU improvements over strong baselines on three of four low-resource languages (aze, glg, slk), with no degradation on the fourth (bel).
• For aze, TCS-S with vocabulary-level similarity achieved 12.37 BLEU, a significant improvement over the best baseline (11.87 BLEU) with p<0.05.
• On bel, TCS-S with vocabulary-level similarity reached 19.83 BLEU, outperforming the best baseline (18.03 BLEU) with p<0.05, while reducing training time.
• In the SDE model setting, TCS achieved 0.5 BLEU gains for aze and up to 2 BLEU for glg and slk, showing strong generalization across architectures.
• TCS-D and TCS-S both outperform the baseline across all languages, with TCS-S showing superior performance due to improved data diversity.
• Development perplexity curves show that TCS consistently reduces loss at a comparable or faster rate than the baseline, indicating effective and efficient training.