[Paper Review] An Actor-Critic Algorithm for Sequence Prediction

Dzmitry Bahdanau, Philémon Brakel+6 more|arXiv (Cornell University)|Jul 24, 2016

Multimodal Machine Learning Applications40 references224 citations

TL;DR

The paper introduces an actor-critic framework for training sequence-generating models, where a critic forecasts token values to improve test-time metrics like BLEU, outperforming MLE and REINFORCE on spelling correction and machine translation tasks.

ABSTRACT

We present an approach to training neural networks to generate sequences using actor-critic methods from reinforcement learning (RL). Current log-likelihood training methods are limited by the discrepancy between their training and testing modes, as models must generate tokens conditioned on their previous guesses rather than the ground-truth tokens. We address this problem by introducing a extit{critic} network that is trained to predict the value of an output token, given the policy of an extit{actor} network. This results in a training procedure that is much closer to the test phase, and allows us to directly optimize for a task-specific score such as BLEU. Crucially, since we leverage these techniques in the supervised learning setting rather than the traditional RL setting, we condition the critic network on the ground-truth output. We show that our method leads to improved performance on both a synthetic task, and for German-English machine translation. Our analysis paves the way for such methods to be applied in natural language generation tasks, such as machine translation, caption generation, and dialogue modelling.

Motivation & Objective

• Motivate training sequence models to optimize task-specific scores rather than only log-likelihood.
• Address the train-test mismatch by conditioning training on model-generated prefixes.
• Introduce a critic network to predict value of each token under the current policy.
• Demonstrate improvements over standard MLE and REINFORCE on spelling correction and machine translation tasks.

Proposed method

• Formulate sequence generation as a stochastic policy with an actor (decoder) and a critic.
• Define value functions V and Q for partial sequences and candidate actions (tokens).
• Train the critic with temporal-difference targets and stabilize with a target network and a delayed actor.
• Use policy gradient with an unbiased estimate incorporating Q estimates, plus optional log-likelihood gradient term.
• Apply reward shaping to provide intermediate feedback and reduce sparse rewards.
• Pre-train both actor and critic before joint actor-critic training to bootstrap learning.

Experimental results

Research questions

• RQ1 Can actor-critic training improve task-specific sequence scores (e.g., BLEU) compared to MLE and REINFORCE?
• RQ2 Does incorporating ground-truth information into the critic aid training without using it at test time?
• RQ3 What training tricks (target networks, reward shaping, value penalties) are essential for stability and performance in sequence prediction?
• RQ4 How does the method perform on synthetic spelling correction and real MT datasets (IWSLT, WMT) relative to baselines?

Key findings

• Actor-critic training yields improvements over log-likelihood training in spelling correction across varied settings.
• On IWSLT 2014 and WMT14 MT tasks, actor-critic methods achieve BLEU gains over baselines, with notable gains in greedy decoding and still competitive with beam search.
• Using a target network and a variance penalty on critic outputs is crucial for stable learning and better performance.
• Reward shaping and delayed actor contribute additional performance gains.
• The approach achieves competitive or superior results compared with prior RL-based methods like MIXER, under stronger or comparable baselines.