[Paper Review] Actor-Critic based Training Framework for Abstractive Summarization

Piji Li, Lidong Bing+1 more|arXiv (Cornell University)|Mar 28, 2018

Topic Modeling31 references46 citations

TL;DR

The paper proposes an actor-critic training framework (AC-ABS) for neural abstractive summarization, combining a seq2seq actor with a maximum likelihood critic and a global quality discriminator, trained in an alternating fashion to improve ROUGE performance across multiple languages.

ABSTRACT

We present a training framework for neural abstractive summarization based on actor-critic approaches from reinforcement learning. In the traditional neural network based methods, the objective is only to maximize the likelihood of the predicted summaries, no other assessment constraints are considered, which may generate low-quality summaries or even incorrect sentences. To alleviate this problem, we employ an actor-critic framework to enhance the training procedure. For the actor, we employ the typical attention based sequence-to-sequence (seq2seq) framework as the policy network for summary generation. For the critic, we combine the maximum likelihood estimator with a well designed global summary quality estimator which is a neural network based binary classifier aiming to make the generated summaries indistinguishable from the human-written ones. Policy gradient method is used to conduct the parameter learning. An alternating training strategy is proposed to conduct the joint training of the actor and critic models. Extensive experiments on some benchmark datasets in different languages show that our framework achieves improvements over the state-of-the-art methods.

Motivation & Objective

• Motivate training of neural abstractive summarization beyond pure likelihood optimization to reduce low-quality outputs (repetitions, noisy tokens, OOV).
• Introduce an actor-critic framework where the actor is an attention-based seq2seq model and critics provide quality signals.
• Demonstrate that alternating training of actor and critics yields superior performance to standard training on multilingual benchmarks.

Proposed method

• Adopt an attention-based seq2seq actor (encoder: bidirectional GRUs; decoder: two-layer GRUs with global attention) as the policy network.
• Critic I uses negative log-likelihood (MLE) as the value function, guiding standard supervised-like updates.
• Critic II introduces a global summary quality estimator (binary classifier) to discriminate generated summaries from human references, guiding policy via REINFORCE with V_phi as reward.
• Use a binary discriminator V_phi that compares representations of X and Y (ground truth) vs generated Y to produce a reward signal.
• Train actor and critics with an alternating strategy (pre-train with Critic I, then alternate updates with Critic I and Critic II, updating Critic II every K3 steps).
• Policy gradient (REINFORCE) is used to propagate the Critic II reward to the actor; critics are learned via gradient-based updates (NLL for Critic I, cross-entropy for Critic II).

Experimental results

Research questions

• RQ1 Can actor-critic training improve abstractive summarization quality beyond traditional maximum-likelihood training?
• RQ2 Does incorporating a global summary quality discriminator help mitigate issues like repetition and noisy outputs in generated summaries?
• RQ3 Is alternating training of actor and critics more stable and effective than joint training or vanilla seq2seq training?
• RQ4 How does AC-ABS perform across English Gigawords, DUC-2004, and Chinese LCSTS datasets in terms of ROUGE metrics?

Key findings

• AC-ABS achieves the best ROUGE scores across reported metrics on Gigawords (R-1, R-2, R-L) compared to baselines including ABS, ABS+, RAS-LSTM, RAS-Elman, ASC-FSC, and others.
• On Gigawords, AC-ABS attains R-1: 36.05, R-2: 17.35, R-L: 33.49, outperforming listed baselines.
• On DUC-2004, AC-ABS achieves ROUGE-Recall R-1: 29.41, R-2: 9.84, R-L: 25.85, surpassing several competitive systems.
• On LCSTS, AC-ABS reaches ROUGE-F1 R-1: 37.51, R-2: 24.68, R-L: 35.02, the best among the compared methods.
• Training analysis shows accelerated convergence and improved ROUGE metrics when using the alternating actor-critic strategy versus standard seq2seq training.
• Critic II achieves very small losses during training, indicating effective discrimination between generated and ground-truth summaries and stabilizing learning.