[Paper Review] Norm-Based Curriculum Learning for Neural Machine Translation

Xuebo Liu, Houtim Lai+2 more|arXiv (Cornell University)|Jun 3, 2020

Natural Language Processing Techniques37 references8 citations

TL;DR

This paper proposes a norm-based curriculum learning method for neural machine translation (NMT) that uses word embedding norms to automatically determine sentence difficulty, model competence, and sentence weight. By leveraging norms as a proxy for linguistic and model-based difficulty, the method enables fully automated curriculum training, achieving +1.17/+1.56 BLEU gains and 2.22x/3.33x training speedup on WMT’14 En-De and WMT’17 Zh-En benchmarks without architectural changes or extra parameters.

ABSTRACT

A neural machine translation (NMT) system is expensive to train, especially with high-resource settings. As the NMT architectures become deeper and wider, this issue gets worse and worse. In this paper, we aim to improve the efficiency of training an NMT by introducing a novel norm-based curriculum learning method. We use the norm (aka length or module) of a word embedding as a measure of 1) the difficulty of the sentence, 2) the competence of the model, and 3) the weight of the sentence. The norm-based sentence difficulty takes the advantages of both linguistically motivated and model-based sentence difficulties. It is easy to determine and contains learning-dependent features. The norm-based model competence makes NMT learn the curriculum in a fully automated way, while the norm-based sentence weight further enhances the learning of the vector representation of the NMT. Experimental results for the WMT'14 English-German and WMT'17 Chinese-English translation tasks demonstrate that the proposed method outperforms strong baselines in terms of BLEU score (+1.17/+1.56) and training speedup (2.22x/3.33x).

Motivation & Objective

• To improve training efficiency and performance of NMT systems, especially in high-resource and deep architecture settings.
• To address the challenge of costly and manual curriculum design in NMT training by automating difficulty assessment and curriculum scheduling.
• To reduce bias in representation learning by introducing norm-based sentence weights that balance learning across sentence difficulties.
• To develop a fully automated, parameter-free curriculum learning framework that adapts to model competence without task-specific hyperparameters.

Proposed method

• Uses the L2 norm of word embeddings as a proxy for sentence difficulty, combining linguistic simplicity (e.g., word frequency) with model-learned features.
• Defines model competence based on the norm of the hidden state representations, enabling automatic curriculum progression as the model improves.
• Introduces sentence weights in the loss function proportional to the inverse of sentence norm, reducing bias toward frequent or low-norm sentences.
• Applies the norm-based curriculum during mini-batch sampling, reordering batches from low to high norm (easy to difficult) throughout training.
• Employs a pre-trained simple word embedding model to compute norms, ensuring low computational cost and expressiveness.
• Integrates the norm-based curriculum into standard NMT training without modifying the model architecture or adding parameters.

Experimental results

Research questions

• RQ1 Can word embedding norms serve as an effective, automated measure of sentence difficulty that combines linguistic and model-based features?
• RQ2 Can model competence be estimated using norm-based metrics to enable fully automated curriculum progression in NMT training?
• RQ3 How does incorporating norm-based sentence weights improve representation learning and model generalization in NMT?
• RQ4 Does the proposed method outperform strong baselines in both translation quality and training efficiency on standard benchmarks?
• RQ5 Can the norm-based curriculum be applied universally across different NMT architectures and language pairs without architectural modifications?

Key findings

• The proposed method achieves a +1.17 BLEU point improvement on the WMT’14 English–German translation task compared to strong baselines.
• On the WMT’17 Chinese–English translation task, the method yields a +1.56 BLEU point gain, demonstrating strong generalization across language pairs.
• The method reduces training time by 2.22x on WMT’14 En-De and 3.33x on WMT’17 Zh-En, indicating significant speedup.
• The norm-based sentence weight mechanism improves the model’s ability to translate complex sentences, as shown in a case study where the vanilla model omitted clauses but the proposed model translated them fully.
• The method achieves superior performance without modifying the NMT architecture or introducing additional parameters, making it practical for real-world deployment.
• The ablation study confirms that combining competence-based curriculum with norm-based sentence weights yields the best balance between learning simple and medium-difficulty sentences.