[Paper Review] Sequence Level Training with Recurrent Neural Networks

Marc’Aurelio Ranzato, Sumit Chopra+2 more|arXiv (Cornell University)|Nov 20, 2015

Topic Modeling29 references945 citations

TL;DR

This paper proposes MIXER, a sequence-level training method for recurrent neural networks that reduces exposure bias and directly optimizes evaluation metrics like BLEU and ROUGE. By combining cross-entropy pre-training with incremental REINFORCE-based policy learning, MIXER achieves state-of-the-art greedy generation performance—outperforming strong baselines and even beam-search-augmented models—while being significantly faster, especially on summarization and image captioning tasks.

ABSTRACT

Many natural language processing applications use language models to generate text. These models are typically trained to predict the next word in a sequence, given the previous words and some context such as an image. However, at test time the model is expected to generate the entire sequence from scratch. This discrepancy makes generation brittle, as errors may accumulate along the way. We address this issue by proposing a novel sequence level training algorithm that directly optimizes the metric used at test time, such as BLEU or ROUGE. On three different tasks, our approach outperforms several strong baselines for greedy generation. The method is also competitive when these baselines employ beam search, while being several times faster.

Motivation & Objective

• To address exposure bias in sequence generation, where models are trained on ground-truth inputs but infer from their own predictions at test time.
• To directly optimize sequence-level evaluation metrics such as BLEU and ROUGE, which are non-differentiable and combinatorially complex.
• To enable effective reinforcement learning in large action spaces typical of text generation, where random policy exploration fails.
• To develop a training method that is both effective and efficient, avoiding the need for expensive beam search at inference.
• To demonstrate that sequence-level training with policy optimization can surpass standard cross-entropy training, even when combined with beam search.

Proposed method

• MIXER uses a hybrid loss combining cross-entropy and REINFORCE to train the model, starting from a cross-entropy pre-trained policy instead of a random one.
• The method employs incremental learning, gradually increasing the proportion of model-generated tokens during training to simulate test-time behavior.
• It applies the REINFORCE algorithm to optimize the expected reward (e.g., BLEU or ROUGE) based on sampled sequences, bypassing the need for differentiable rewards.
• The model is trained using a single sample per step to estimate the policy gradient, with a baseline for variance reduction.
• The approach is model-agnostic and can be applied to any sequence model, including RNNs, with any differentiable reward function.
• The training schedule is tuned via grid search to balance exploration and exploitation, ensuring convergence.

Experimental results

Research questions

• RQ1 Can sequence-level training with reinforcement learning improve text generation quality compared to standard next-word prediction?
• RQ2 Does replacing random policy initialization with a cross-entropy pre-trained policy enable stable training in large action spaces?
• RQ3 Can MIXER outperform strong baselines like cross-entropy with beam search in both performance and inference speed?
• RQ4 How effective is the combination of incremental learning and hybrid loss in reducing exposure bias?
• RQ5 Is direct optimization of metrics like BLEU and ROUGE feasible and beneficial in practice?

Key findings

• On summarization, MIXER with greedy generation achieved a ROUGE-2 score of 16.22, outperforming the XENT baseline (13.01) and DAD (12.18).
• In machine translation, MIXER achieved a BLEU-4 score of 20.73, surpassing the XENT baseline (17.74) and DAD (20.12).
• For image captioning, MIXER reached a BLEU-4 score of 29.16, exceeding XENT (27.8) and DAD (28.16).
• MIXER with greedy search outperformed XENT with beam search (k=10) on two of the three tasks, demonstrating superior efficiency and performance.
• The method is at least 10 times faster than beam search with k=10, while maintaining or exceeding performance on all tasks.
• Combining MIXER with beam search further improves results, but the gains are task-dependent, indicating that MIXER and beam search are complementary.