[Paper Review] Challenges in Data-to-Document Generation

Sam Wiseman, Stuart M. Shieber+1 more|arXiv (Cornell University)|Jul 25, 2017

Topic Modeling32 references53 citations

TL;DR

The paper introduces a large NBA box-score dataset for data-to-document generation, proposes extractive evaluation metrics, and benchmarks neural and templated baselines, showing neural models struggle with content selection and long-range structure compared to templates.

ABSTRACT

Recent neural models have shown significant progress on the problem of generating short descriptive texts conditioned on a small number of database records. In this work, we suggest a slightly more difficult data-to-text generation task, and investigate how effective current approaches are on this task. In particular, we introduce a new, large-scale corpus of data records paired with descriptive documents, propose a series of extractive evaluation methods for analyzing performance, and obtain baseline results using current neural generation methods. Experiments show that these models produce fluent text, but fail to convincingly approximate human-generated documents. Moreover, even templated baselines exceed the performance of these neural models on some metrics, though copy- and reconstruction-based extensions lead to noticeable improvements.

Motivation & Objective

• Motivate a longer, document-scale data-to-text task that emphasizes fidelity to structured data.
• Provide a publicly available dataset pairing basketball game records with descriptive documents.
• Propose automatic extractive evaluation metrics to assess content selection, generation, and ordering.
• Baseline neural and templated generation methods to establish performance benchmarks for long-form data-to-document generation.

Proposed method

• Construct a large-scale dataset of basketball game box scores paired with descriptive game reports from two sources (RotoWire and SBNation).
• Represent each data record with embeddings of type, entity, and value and feed these into an attention-based encoder-decoder model.
• Incorporate copy mechanisms to allow direct copying of values from the data records during generation.
• Introduce reconstruction-based loss terms to encourage decoder representations to cover the input data.
• Develop extractive evaluation metrics that measure Content Selection, Relation Generation, and Content Ordering using a relation extractor.
• Compare neural generation approaches (with and without copying and reconstruction) to a templated generator.

Experimental results

Research questions

• RQ1 Can a data-to-document model generate faithful, multi-sentence sports summaries conditioned on box-score records?
• RQ2 How do neural generation methods compare to templated baselines in terms of content fidelity and overall fluency?
• RQ3 Do extractive evaluation metrics align with human judgments of factual accuracy and content coverage?
• RQ4 What impact do copying and reconstruction losses have on content selection and long-range discourse?
• RQ5 What are the limitations of current neural approaches for data-driven, document-level generation?

Key findings

• Neural models produce fluent text but struggle with accurate content selection and long-term structure.
• Copy-based and reconstruction-enhanced models improve BLEU and extractive metrics but remain far from human-level output.
• Templated generation can outperform neural models on some metrics, highlighting content selection gaps in neural approaches.
• An extractive evaluation framework reveals fidelity issues hidden by standard automatic metrics.
• Conditional copy and reconstruction strategies offer the most promise among neural baselines, yet still lag behind gold data.