[Paper Review] Evaluating the Factual Consistency of Abstractive Text Summarization

Wojciech Kryściński, Bryan McCann+2 more|arXiv (Cornell University)|Oct 28, 2019

Topic Modeling4 citations

TL;DR

This paper proposes a weakly-supervised, BERT-based model for evaluating factual consistency in abstractive text summarization by generating training data through rule-based transformations of source documents. The model jointly predicts consistency, extracts supporting spans from the source, and identifies inconsistent spans in summaries—outperforming strong supervision baselines and demonstrating human-assistive utility through explainable span highlighting.

ABSTRACT

Currently used metrics for assessing summarization algorithms do not account for whether summaries are factually consistent with source documents. We propose a weakly-supervised, model-based approach for verifying factual consistency and identifying conflicts between source documents and a generated summary. Training data is generated by applying a series of rule-based transformations to the sentences of source documents. The factual consistency model is then trained jointly for three tasks: 1) identify whether sentences remain factually consistent after transformation, 2) extract a span in the source documents to support the consistency prediction, 3) extract a span in the summary sentence that is inconsistent if one exists. Transferring this model to summaries generated by several state-of-the art models reveals that this highly scalable approach substantially outperforms previous models, including those trained with strong supervision using standard datasets for natural language inference and fact checking. Additionally, human evaluation shows that the auxiliary span extraction tasks provide useful assistance in the process of verifying factual consistency.

Motivation & Objective

• To address the lack of evaluation protocols that verify factual consistency in abstractive summarization.
• To develop a scalable, weakly-supervised method for detecting factual inconsistencies between source documents and generated summaries.
• To improve factual consistency evaluation by incorporating explainable span extraction for source and summary.
• To demonstrate that model-generated highlights assist human annotators in factual consistency verification.

Proposed method

• Training data is synthesized via rule-based transformations applied to source document sentences to simulate factual inconsistencies.
• A multi-task BERT-based model is trained to predict factual consistency, extract supporting spans from the source, and identify inconsistent spans in the summary.
• The model uses weak supervision derived from error analysis of state-of-the-art summarization models, avoiding reliance on costly human-annotated entailment or fact-checking datasets.
• Span extraction components are trained jointly with the consistency prediction to provide interpretable rationales for decisions.
• The model is fine-tuned on summarization outputs and evaluated via human annotation and automatic metrics.
• Human evaluation compares annotation speed and inter-annotator agreement with and without model-provided highlights.

Experimental results

Research questions

• RQ1 Can a weakly-supervised approach based on synthetic data outperform models trained on strong supervision from NLI and fact-checking datasets?
• RQ2 To what extent do explainable span predictions improve human performance in factual consistency verification?
• RQ3 How effective is the model at identifying both consistent and inconsistent claims in abstractive summaries?
• RQ4 What impact do model-generated highlights have on the efficiency and reliability of human annotation?

Key findings

• The FactCCX model outperforms strong supervision baselines trained on NLI and fact-checking datasets in factual consistency detection.
• Human annotators completed factual consistency tasks 21% faster when assisted by model-generated highlights.
• Inter-annotator agreement (Fleiss’ κ) increased by 38% when using model-provided highlights, indicating improved annotation reliability.
• The model achieved 65.33% and 65.66% accuracy in highlighting relevant spans in source documents and summaries, respectively.
• The F1 score for span overlap reached 0.6207 (source) and 0.6650 (claim), showing strong alignment with human-annotated spans.
• Human evaluation confirmed that the auxiliary span extraction tasks significantly assist in verifying factual consistency.