[Paper Review] Boosting Neural Machine Translation

Dakun Zhang, Jun-Gi Kim+2 more|arXiv (Cornell University)|Dec 19, 2016

Natural Language Processing Techniques19 references16 citations

TL;DR

This paper proposes a data boosting approach for Neural Machine Translation (NMT) that improves training efficiency and translation accuracy by dynamically reweighting training data based on instance difficulty, measured via perplexity. By focusing training on the most challenging sentence pairs—selecting only the top 80% by perplexity in each epoch—the method achieves a 1.63 BLEU improvement while reducing training time by 20%, without modifying the neural network architecture.

ABSTRACT

Training efficiency is one of the main problems for Neural Machine Translation (NMT). Deep networks need for very large data as well as many training iterations to achieve state-of-the-art performance. This results in very high computation cost, slowing down research and industrialisation. In this paper, we propose to alleviate this problem with several training methods based on data boosting and bootstrap with no modifications to the neural network. It imitates the learning process of humans, which typically spend more time when learning "difficult" concepts than easier ones. We experiment on an English-French translation task showing accuracy improvements of up to 1.63 BLEU while saving 20% of training time.

Motivation & Objective

• To address the high computational cost and long training times associated with state-of-the-art NMT systems.
• To improve training efficiency and model convergence stability without modifying the neural network architecture.
• To explore whether prioritizing 'difficult' training examples leads to better generalization and faster convergence.
• To develop a data-centric training policy that mimics human learning by allocating more training time to challenging instances.

Proposed method

• The method selects the top 80% of training instances with the highest perplexity values after each training epoch to form the next training batch.
• Perplexity is computed per training batch and used as a proxy for instance difficulty, with no normalization applied in the final experiments.
• The approach operates without modifying the neural network, making it compatible with any NMT framework and training setup.
• The training policy dynamically reweights data over time, emphasizing difficult examples while reducing reliance on easier ones.
• The method is applied in a data-boosting fashion, where the model is trained on a filtered subset of the most challenging examples in each epoch.
• The approach is evaluated using standard BLEU scores and training time metrics across multiple training policies, including baseline, boosting, reduction, and bootstrapping.

Experimental results

Research questions

• RQ1 Can reweighting training data based on instance difficulty improve NMT performance while reducing training time?
• RQ2 Does focusing training on high-perplexity (difficult) examples lead to faster convergence and better generalization?
• RQ3 How does the size of the training subset (e.g., 80% vs. 100%) affect translation quality and training efficiency?
• RQ4 Is the performance gain stable across different training epochs and initialization seeds?
• RQ5 Can this method be applied universally across different NMT architectures and training frameworks without architectural modifications?

Key findings

• The proposed method achieved a 1.63 BLEU point improvement over the baseline system while reducing training data usage by 20%.
• The 'reduce' policy, which uses only the 80% most difficult examples per epoch, outperformed the baseline and other policies in both accuracy and efficiency.
• The system converged faster, reaching peak performance at epoch 14 compared to epoch 16 for the baseline, indicating improved training stability.
• The 'boost' policy, which added 10% more data, improved BLEU by 1.49 points, showing consistent gains with minimal data increase.
• The 'bootstrap' policy, which reuses the full dataset with dynamic selection, achieved a 0.87 BLEU improvement with stable performance over time.
• Ensembling the best models from all four training policies yielded an additional 0.92 BLEU improvement, reaching a final score of 55.04.