[论文解读] Neural Abstractive Text Summarization with Sequence-to-Sequence Models

Tian Shi, Yaser Keneshloo+2 more|arXiv (Cornell University)|Dec 5, 2018

Topic Modeling参考文献 154被引用 68

一句话总结

一份关于基于 seq2seq 的神经网络抽象文本摘要的综合综述，涵盖网络结构、训练策略和生成方法，并提供开源的 NATS 工具包，以及在 CNN/Daily Mail、Newsroom 和 Bytecup 上的实验。

ABSTRACT

In the past few years, neural abstractive text summarization with sequence-to-sequence (seq2seq) models have gained a lot of popularity. Many interesting techniques have been proposed to improve seq2seq models, making them capable of handling different challenges, such as saliency, fluency and human readability, and generate high-quality summaries. Generally speaking, most of these techniques differ in one of these three categories: network structure, parameter inference, and decoding/generation. There are also other concerns, such as efficiency and parallelism for training a model. In this paper, we provide a comprehensive literature survey on different seq2seq models for abstractive text summarization from the viewpoint of network structures, training strategies, and summary generation algorithms. Several models were first proposed for language modeling and generation tasks, such as machine translation, and later applied to abstractive text summarization. Hence, we also provide a brief review of these models. As part of this survey, we also develop an open source library, namely, Neural Abstractive Text Summarizer (NATS) toolkit, for the abstractive text summarization. An extensive set of experiments have been conducted on the widely used CNN/Daily Mail dataset to examine the effectiveness of several different neural network components. Finally, we benchmark two models implemented in NATS on the two recently released datasets, namely, Newsroom and Bytecup.

研究动机与目标

• 概述跨网络结构和训练策略的 seq2seq 抽象文本摘要模型现状。
• 评述关键机制，如注意力、复制机制，以及处理长文档以提升显著性、流畅性和可读性。
• 提供一个开源工具包（NATS）和关于标准数据集的基准见解，以促进复现和进一步研究。

提出的方法

• 综述用于抽象摘要的基础 seq2seq 架构和注意力机制。
• 讨论指针生成网络和复制机制，以解决 OOV 词和事实准确性。
• 解释包括课程学习和强化学习在内的训练策略，以缓解暴露偏差和指标错位。
• 总结在效率和性能方面改进的 CNN/卷积 seq2seq 与 Transformer 架构的发展。
• 提供一个开源库（NATS）并报告在 CNN/Daily Mail、Newsroom 和 Bytecup 数据集上的实验。

实验结果

研究问题

• RQ1 哪些主要网络结构与组件使 seq2seq 模型能够实现高质量的抽象摘要？
• RQ2 训练策略和解码算法如何解决暴露偏差、目标错位和生成质量问题？
• RQ3 在标准基准上，不同架构（RNN 基础、CNN 基础、Transformer）对抽象摘要有哪些实证证据？
• RQ4 如何通过开源工具和标准化实验来提升可复现性？

主要发现

• 该综述将 seq2seq 抽象摘要分为网络结构、训练策略和生成算法。
• 指针生成和复制机制在处理 OOV 词和事实内容方面有所提升。
• 以课程学习为特征的 RL 训练有助于使训练目标与非微分评估指标（如 ROUGE）对齐。
• 基于 CNN 和 Transformer 的架构在效率和性能上与基于 RNN 的模型相比具有竞争力。
• 提供一个开源的 NATS 工具包，用于复现和扩展摘要模型，并在 CNN/Daily Mail、Newsroom、Bytecup 上进行实验。