[Paper Review] Boosting Factual Correctness of Abstractive Summarization with Knowledge Graph.

Chenguang Zhu, William Hinthorn+5 more|arXiv (Cornell University)|Mar 19, 2020

Topic Modeling34 citations

TL;DR

This paper proposes FASum, a fact-aware abstractive summarization model that enhances factual correctness by integrating knowledge graphs derived from input text. By leveraging neural graph computation to fuse factual relations into the generation process, FASum achieves 1.2% higher factual correctness than UniLM and 4.5% higher than BottomUp on the CNN/DailyMail dataset, significantly outperforming state-of-the-art models.

ABSTRACT

A commonly observed problem with abstractive summarization is the distortion or fabrication of factual information in the article. This inconsistency between summary and original text has led to various concerns over its applicability. In this paper, we propose to boost factual correctness of summaries via the fusion of knowledge, i.e. extracted factual relations from the article. We present a Fact-Aware Summarization model, FASum. In this model, the knowledge information can be organically integrated into the summary generation process via neural graph computation and effectively improves the factual correctness. Empirical results show that FASum generates summaries with significantly higher factual correctness compared with state-of-the-art abstractive summarization systems, both under an independently trained factual correctness evaluator and human evaluation. For example, in CNN/DailyMail dataset, FASum obtains 1.2% higher fact correctness scores than UniLM and 4.5% higher than BottomUp.

Motivation & Objective

• To address the persistent issue of factual hallucinations in abstractive summarization.
• To improve factual correctness in generated summaries without relying solely on sequence modeling.
• To integrate factual knowledge extracted from input text into the summarization process in a structured, learnable way.
• To develop a neural graph-based mechanism that enhances factual consistency during summary generation.
• To demonstrate superior factual correctness over state-of-the-art abstractive models through both automatic and human evaluation.

Proposed method

• The model constructs a knowledge graph from factual relations extracted from the input article using relation extraction techniques.
• Factual relations are encoded as nodes and edges in a graph structure, representing entities and their relationships.
• Neural graph computation is applied to propagate and aggregate information across the graph, enriching contextual representations.
• The graph-enhanced representations are fused into the decoder of a sequence-to-sequence model to guide factually grounded summary generation.
• The model is trained end-to-end with a combination of reconstruction and generation objectives to preserve factual consistency.
• Knowledge graph attention mechanisms are used to dynamically attend to relevant facts during decoding.

Experimental results

Research questions

• RQ1 Can integrating structured factual knowledge from input text improve factual correctness in abstractive summarization?
• RQ2 How does graph-based knowledge integration compare to standard sequence modeling in reducing factual hallucinations?
• RQ3 To what extent does the proposed method outperform state-of-the-art models in factual correctness under automatic and human evaluation?
• RQ4 Does the use of neural graph computation enhance the model's ability to preserve factual consistency during summary generation?
• RQ5 Can knowledge graph fusion lead to measurable improvements in factual correctness metrics on benchmark datasets?

Key findings

• FASum achieves 1.2% higher factual correctness scores than UniLM on the CNN/DailyMail dataset under an independent factual correctness evaluator.
• FASum outperforms BottomUp by 4.5% in factual correctness on the same benchmark, demonstrating significant improvement.
• Human evaluation confirms that FASum generates summaries with higher factual consistency compared to state-of-the-art models.
• The integration of knowledge graphs via neural graph computation leads to more accurate and reliable summary generation.
• The model’s performance gains are consistent across both automatic and human evaluation, validating its effectiveness.
• Factual relations extracted from the input are effectively leveraged during decoding to reduce hallucinations and improve factual fidelity.