[论文解读] Actor-Critic based Training Framework for Abstractive Summarization

Piji Li, Lidong Bing+1 more|arXiv (Cornell University)|Mar 28, 2018

Topic Modeling参考文献 31被引用 46

一句话总结

该论文提出一种 actor-critic 训练框架（AC-ABS）用于神经抽象摘要，将 seq2seq actor 与最大似然 critic 以及全局质量判别器结合，在交替训练中提升跨多语言的 ROUGE 性能。

ABSTRACT

We present a training framework for neural abstractive summarization based on actor-critic approaches from reinforcement learning. In the traditional neural network based methods, the objective is only to maximize the likelihood of the predicted summaries, no other assessment constraints are considered, which may generate low-quality summaries or even incorrect sentences. To alleviate this problem, we employ an actor-critic framework to enhance the training procedure. For the actor, we employ the typical attention based sequence-to-sequence (seq2seq) framework as the policy network for summary generation. For the critic, we combine the maximum likelihood estimator with a well designed global summary quality estimator which is a neural network based binary classifier aiming to make the generated summaries indistinguishable from the human-written ones. Policy gradient method is used to conduct the parameter learning. An alternating training strategy is proposed to conduct the joint training of the actor and critic models. Extensive experiments on some benchmark datasets in different languages show that our framework achieves improvements over the state-of-the-art methods.

研究动机与目标

• Motivate training of neural abstractive summarization beyond pure likelihood optimization to reduce low-quality outputs (repetitions, noisy tokens, OOV).
• Introduce an actor-critic framework where the actor is an attention-based seq2seq model and critics provide quality signals.
• Demonstrate that alternating training of actor and critics yields superior performance to standard training on multilingual benchmarks.

提出的方法

• Adopt an attention-based seq2seq actor (encoder: bidirectional GRUs; decoder: two-layer GRUs with global attention) as the policy network.
• Critic I uses negative log-likelihood (MLE) as the value function, guiding standard supervised-like updates.
• Critic II introduces a global summary quality estimator (binary classifier) to discriminate generated summaries from human references, guiding policy via REINFORCE with V_phi as reward.
• Use a binary discriminator V_phi that compares representations of X and Y (ground truth) vs generated Y to produce a reward signal.
• Train actor and critics with an alternating strategy (pre-train with Critic I, then alternate updates with Critic I and Critic II, updating Critic II every K3 steps).
• Policy gradient (REINFORCE) is used to propagate the Critic II reward to the actor; critics are learned via gradient-based updates (NLL for Critic I, cross-entropy for Critic II).

实验结果

研究问题

• RQ1 Can actor-critic training improve abstractive summarization quality beyond traditional maximum-likelihood training?
• RQ2 Does incorporating a global summary quality discriminator help mitigate issues like repetition and noisy outputs in generated summaries?
• RQ3 Is alternating training of actor and critics more stable and effective than joint training or vanilla seq2seq training?
• RQ4 How does AC-ABS perform across English Gigawords, DUC-2004, and Chinese LCSTS datasets in terms of ROUGE metrics?

主要发现

• AC-ABS achieves the best ROUGE scores across reported metrics on Gigawords (R-1, R-2, R-L) compared to baselines including ABS, ABS+, RAS-LSTM, RAS-Elman, ASC-FSC, and others.
• On Gigawords, AC-ABS attains R-1: 36.05, R-2: 17.35, R-L: 33.49, outperforming listed baselines.
• On DUC-2004, AC-ABS achieves ROUGE-Recall R-1: 29.41, R-2: 9.84, R-L: 25.85, surpassing several competitive systems.
• On LCSTS, AC-ABS reaches ROUGE-F1 R-1: 37.51, R-2: 24.68, R-L: 35.02, the best among the compared methods.
• Training analysis shows accelerated convergence and improved ROUGE metrics when using the alternating actor-critic strategy versus standard seq2seq training.
• Critic II achieves very small losses during training, indicating effective discrimination between generated and ground-truth summaries and stabilizing learning.