[论文解读] Challenges in Data-to-Document Generation

Sam Wiseman, Stuart M. Shieber+1 more|arXiv (Cornell University)|Jul 25, 2017

Topic Modeling参考文献 32被引用 53

一句话总结

本文为数据到文档生成引入了一个大型的 NBA box-score 数据集，提出了 extractive 评估指标，并对神经模型与模板基线进行基准测试，结果表明神经模型在内容选择和长程结构方面相较模板存在困难。

ABSTRACT

Recent neural models have shown significant progress on the problem of generating short descriptive texts conditioned on a small number of database records. In this work, we suggest a slightly more difficult data-to-text generation task, and investigate how effective current approaches are on this task. In particular, we introduce a new, large-scale corpus of data records paired with descriptive documents, propose a series of extractive evaluation methods for analyzing performance, and obtain baseline results using current neural generation methods. Experiments show that these models produce fluent text, but fail to convincingly approximate human-generated documents. Moreover, even templated baselines exceed the performance of these neural models on some metrics, though copy- and reconstruction-based extensions lead to noticeable improvements.

研究动机与目标

• Motivate a longer, document-scale data-to-text task that emphasizes fidelity to structured data.
• Provide a publicly available dataset pairing basketball game records with descriptive documents.
• Propose automatic extractive evaluation metrics to assess content selection, generation, and ordering.
• Baseline neural and templated generation methods to establish performance benchmarks for long-form data-to-document generation.

提出的方法

• Construct a large-scale dataset of basketball game box scores paired with descriptive game reports from two sources (RotoWire and SBNation).
• Represent each data record with embeddings of type, entity, and value and feed these into an attention-based encoder-decoder model.
• Incorporate copy mechanisms to allow direct copying of values from the data records during generation.
• Introduce reconstruction-based loss terms to encourage decoder representations to cover the input data.
• Develop extractive evaluation metrics that measure Content Selection, Relation Generation, and Content Ordering using a relation extractor.
• Compare neural generation approaches (with and without copying and reconstruction) to a templated generator.

实验结果

研究问题

• RQ1 Can a data-to-document model generate faithful, multi-sentence sports summaries conditioned on box-score records?
• RQ2 How do neural generation methods compare to templated baselines in terms of content fidelity and overall fluency?
• RQ3 Do extractive evaluation metrics align with human judgments of factual accuracy and content coverage?
• RQ4 What impact do copying and reconstruction losses have on content selection and long-range discourse?
• RQ5 What are the limitations of current neural approaches for data-driven, document-level generation?

主要发现

• Neural models produce fluent text but struggle with accurate content selection and long-term structure.
• Copy-based and reconstruction-enhanced models improve BLEU and extractive metrics but remain far from human-level output.
• Templated generation can outperform neural models on some metrics, highlighting content selection gaps in neural approaches.
• An extractive evaluation framework reveals fidelity issues hidden by standard automatic metrics.
• Conditional copy and reconstruction strategies offer the most promise among neural baselines, yet still lag behind gold data.