[Paper Review] Analyzing Sentence Fusion in Abstractive Summarization

Logan Lebanoff, John Muchovej+5 more|arXiv (Cornell University)|Oct 1, 2019

Topic Modeling34 references4 citations

TL;DR

This paper analyzes sentence fusion in abstractive summarization by evaluating five state-of-the-art models on CNN/DailyMail using human annotations. It finds that 38.3% of fused summary sentences introduce incorrect facts, with entity replacement and complex fusion methods being particularly prone to errors, while simple concatenation yields more faithful and grammatical outputs.

ABSTRACT

While recent work in abstractive summarization has resulted in higher scores in automatic metrics, there is little understanding on how these systems combine information taken from multiple document sentences. In this paper, we analyze the outputs of five state-of-the-art abstractive summarizers, focusing on summary sentences that are formed by sentence fusion. We ask assessors to judge the grammaticality, faithfulness, and method of fusion for summary sentences. Our analysis reveals that system sentences are mostly grammatical, but often fail to remain faithful to the original article.

Motivation & Objective

• To investigate how state-of-the-art abstractive summarizers perform sentence fusion, a core challenge in content consolidation.
• To evaluate the faithfulness, grammaticality, and fusion methods used in summary sentences formed by combining multiple source sentences.
• To identify which fusion techniques—such as concatenation, replacement, or complex merging—lead to higher accuracy and grammatical correctness.
• To compare the performance of diverse models (PG, Novel, Fast-Abs-RL, Bottom-Up, DCA) in generating faithful and grammatical fused sentences.
• To provide a human evaluation benchmark for sentence fusion, addressing limitations of automatic metrics like ROUGE.

Proposed method

• Conducted human evaluation via Amazon Mechanical Turk to assess faithfulness, grammaticality, and fusion method for summary sentences.
• Focused on sentences formed by sentence fusion, distinguishing them from compression, copying, or failed matching.
• Used an automatic heuristic to identify source sentence pairs, with coverage measured as the percentage of correct source pairs identified.
• Classified fusion methods into categories: balanced/imbalanced concatenation, replacement, and other complex methods.
• Analyzed model outputs across five SOTA abstractive summarizers, including PG, Novel, Fast-Abs-RL, Bottom-Up, and DCA.
• Employed four annotators per example to improve reliability, given the challenge of judging faithfulness and grammaticality.

Experimental results

Research questions

• RQ1 How often do abstractive summarizers produce ungrammatical or factually incorrect sentences when fusing multiple source sentences?
• RQ2 Which sentence fusion techniques—concatenation, replacement, or complex merging—result in higher faithfulness and grammaticality?
• RQ3 How does the performance of different state-of-the-art models vary in terms of factual consistency and grammatical correctness during fusion?
• RQ4 To what extent does the automatic heuristic for identifying source sentence pairs align with human judgment?
• RQ5 Why do complex fusion methods like entity replacement lead to higher error rates compared to simpler methods like concatenation?

Key findings

• 38.3% of summary sentences generated by state-of-the-art models contain incorrect facts, indicating a major challenge in factual consistency during sentence fusion.
• 21.6% of summary sentences are ungrammatical, with the lowest grammaticality observed in the 'Other' fusion category (68.23%).
• Concatenation-based fusion methods (balanced and imbalanced) achieve the highest faithfulness (82.55% and 69.40%) and grammaticality (86.91% and 80.25%), respectively.
• Entity replacement and complex fusion methods result in incorrect facts 47–75% of the time, indicating high error rates in advanced fusion techniques.
• The coverage of the automatic heuristic for identifying source sentence pairs is 77.3% on average, but lower in complex cases, affecting annotation reliability.
• Despite high ROUGE scores, the PG model is the most faithful system (82.55% faithful for balanced concatenation), suggesting ROUGE alone is insufficient for evaluating faithfulness.