[Paper Review] An Empirical Exploration of Curriculum Learning for Neural Machine Translation

Xuan Zhang, Gaurav Kumar+7 more|arXiv (Cornell University)|Nov 2, 2018

Natural Language Processing Techniques20 references106 citations

TL;DR

The paper investigates a probabilistic curriculum learning framework for German–English NMT, showing potential convergence speedups with no BLEU loss, but results are highly sensitive to curriculum design and hyperparameters.

ABSTRACT

Machine translation systems based on deep neural networks are expensive to train. Curriculum learning aims to address this issue by choosing the order in which samples are presented during training to help train better models faster. We adopt a probabilistic view of curriculum learning, which lets us flexibly evaluate the impact of curricula design, and perform an extensive exploration on a German-English translation task. Results show that it is possible to improve convergence time at no loss in translation quality. However, results are highly sensitive to the choice of sample difficulty criteria, curriculum schedule and other hyperparameters.

Motivation & Objective

• Motivate reducing training time for data-hungry neural machine translation systems by exploring curriculum learning.
• Propose a probabilistic curriculum framework that selects training samples with phase-dependent probabilities.
• Assess a wide range of difficulty criteria and schedules to understand their impact on convergence and BLEU.
• Evaluate curriculum learning on German–English TED Talks data using the Sockeye NMT toolkit.
• Identify practical guidance and limitations for designing curricula in NMT settings.

Proposed method

• Adopt a probabilistic view where each training sample has a time-dependent probability of being selected (q_i^t) to form curricula.
• Organize data into shards of similar difficulty to simplify scheduling and sampling.
• Define difficulty criteria including model-based (one-best score p(y_hat|x)) and linguistic features (sentence length, word-frequency ranks).
• Instantiate several curriculum schedules (default/easy-to-hard, reverse, boost, reduce, noshuffle) to study their effects across epochs or phases.
• Train NMT models in Sockeye on German–English TED Talks data, with 512-d embeddings, LSTM encoder/decoder, and BPE-30k, comparing curriculum variants to a standard baseline.
• Update the curriculum after a fixed number of batches and continue training until validation perplexity stabilizes.

Experimental results

Research questions

• RQ1 Can curriculum learning speed up convergence of NMT models without sacrificing translation quality (BLEU)?
• RQ2 Which difficulty criteria (model-based vs linguistic) and which curriculum schedules best improve convergence under different hyperparameters?
• RQ3 How sensitive are curriculum outcomes to hyperparameters such as learning rate and curriculum update frequency?

Key findings

• Curriculum learning can improve convergence speed without BLEU loss for some configurations; 20 of 100 curricula converged earlier than baseline with no BLEU loss.
• The effect of curricula is highly sensitive to hyperparameters; no single strategy consistently dominates across settings.
• For average word-frequency-based difficulty and a reverse schedule, training time improved by about 19%–30%.
• With a higher learning rate (0.002), the one-best score with boost converged 19% faster (59k vs 73k batches) with BLEU 28.4 vs 28.1; at lr=0.0008, some curricula matched or exceeded baseline BLEU with varying speedups.
• Sentence-length-based criteria offered limited or early-stage benefits and did not consistently improve convergence time at convergence.
• Overall, simple length- or frequency-based criteria can perform as well as more expensive ones, and there is no universally best curriculum; hyperparameters strongly influence outcomes.