[Paper Review] Learning a Multi-Domain Curriculum for Neural Machine Translation

Wei Wang, Ye Tian+4 more|arXiv (Cornell University)|Aug 28, 2019

Natural Language Processing Techniques45 references4 citations

TL;DR

This paper proposes a multi-domain curriculum learning framework for neural machine translation that dynamically selects and prioritizes training data based on instance-level domain relevance and noise reduction. By learning a curriculum that gradually emphasizes high-quality, multi-domain-relevant examples, the method achieves superior performance across all domains—including out-of-domain—while outperforming both single-domain and no-curriculum baselines on large-scale, noisy datasets.

ABSTRACT

Most data selection research in machine translation focuses on improving a single domain. We perform data selection for multiple domains at once. This is achieved by carefully introducing instance-level domain-relevance features and automatically constructing a training curriculum to gradually concentrate on multi-domain relevant and noise-reduced data batches. Both the choice of features and the use of curriculum are crucial for balancing and improving all domains, including out-of-domain. In large-scale experiments, the multi-domain curriculum simultaneously reaches or outperforms the individual performance and brings solid gains over no-curriculum training.

Motivation & Objective

• To address the challenge of training a single NMT model that generalizes well across multiple domains simultaneously.
• To overcome catastrophic forgetting and data imbalance issues in multi-domain NMT training.
• To develop a dynamic data selection method that prioritizes both domain relevance and data quality.
• To improve model performance on both in-domain and out-of-domain test sets without relying on domain labels during training.
• To demonstrate that curriculum learning with instance-level features outperforms static data mixing and loss weighting.

Proposed method

• The method introduces instance-level features that measure domain relevance and data quality for each training example.
• It formulates a multi-domain curriculum by learning to sort training examples based on a weighted combination of domain relevance and noise-reduction scores.
• The curriculum is learned end-to-end using a differentiable objective that balances domain coverage and data quality.
• It uses NMT and NLM features to estimate instance-level relevance and noise, with a threshold-based selection mechanism to form data batches.
• The approach combines dynamic data selection with curriculum learning, enabling gradual model focus on high-quality, multi-domain-relevant examples.
• It supports both training with curriculum and fine-tuning on in-domain data, maintaining gains across domains.

Experimental results

Research questions

• RQ1 Can a single NMT model be trained effectively across multiple domains without suffering from catastrophic forgetting?
• RQ2 How can data selection be made dynamic and instance-specific to improve multi-domain generalization?
• RQ3 Does curriculum learning based on domain relevance and noise reduction outperform static data mixing or loss weighting?
• RQ4 Can the model maintain strong performance on out-of-domain data while improving in-domain accuracy?
• RQ5 How do different feature representations (per-domain vs. multi-domain) affect curriculum learning performance?

Key findings

• The multi-domain curriculum method achieves or exceeds the performance of individually optimized models on each domain, with a 39.2 BLEU score on the average of multiple domains.
• The method brings significant gains over no-curriculum training, improving average BLEU from 36.3 to 39.2 on noisy datasets.
• Curriculum learning with learned weights outperforms simple instance loss weighting, especially on noisy data, with a 3.9 BLEU improvement on the D15 test set.
• Fine-tuning the final curriculum model on in-domain data further improves performance, reaching 40.8 BLEU on average, surpassing baseline fine-tuned models.
• Using a joint multi-domain feature (Eq. 8) slightly outperforms linear combinations of per-domain features, achieving 39.5 BLEU vs. 39.2 BLEU.
• The curriculum dynamically increases focus on high-quality, multi-domain-relevant examples, as shown by feature score trends in Figure 4.