[Paper Review] Asking and Answering Questions to Evaluate the Factual Consistency of Summaries

Alex Wang, Kyunghyun Cho+1 more|arXiv (Cornell University)|Apr 8, 2020

Topic Modeling40 references32 citations

TL;DR

QAGS evaluates factual consistency of abstractive summaries by generating questions from the summary and answering them from both the source article and the summary, then comparing answers. It shows higher correlation with human judgments than standard metrics like ROUGE and BLEU.

ABSTRACT

Practical applications of abstractive summarization models are limited by frequent factual inconsistencies with respect to their input. Existing automatic evaluation metrics for summarization are largely insensitive to such errors. We propose an automatic evaluation protocol called QAGS (pronounced "kags") that is designed to identify factual inconsistencies in a generated summary. QAGS is based on the intuition that if we ask questions about a summary and its source, we will receive similar answers if the summary is factually consistent with the source. To evaluate QAGS, we collect human judgments of factual consistency on model-generated summaries for the CNN/DailyMail (Hermann et al., 2015) and XSUM (Narayan et al., 2018) summarization datasets. QAGS has substantially higher correlations with these judgments than other automatic evaluation metrics. Also, QAGS offers a natural form of interpretability: The answers and questions generated while computing QAGS indicate which tokens of a summary are inconsistent and why. We believe QAGS is a promising tool in automatically generating usable and factually consistent text.

Motivation & Objective

• Motivate the need for evaluating factual consistency beyond n-gram overlap in abstractive summarization.
• Propose a framework that uses question generation and question answering to assess factual alignment with the source text.
• Develop QAGS, a metric that correlates with human judgments of factuality and offers interpretability through its QA prompts.
• Demonstrate robustness of QAGS to underlying model quality, domain shifts, and the number of questions.
• Provide ablations and error analysis to guide future improvements and potential extensions to other modalities.

Proposed method

• Define a three-step evaluation framework: generate questions from the summary, answer questions using both the source and the summary, and compare the corresponding answers.
• Instantiate the framework for QAGS with: (a) question generation conditioned on the summary, (b) extractive QA models to produce answer distributions, and (c) token-level F1 as the answer similarity metric.
• Use beam search to produce a set of high-probability questions and filter for quality with heuristics and QA consistency checks.
• Compute the QAGS score by averaging the similarity of corresponding answers across the selected questions.
• Report correlations with human judgments on CNN/DailyMail and XSUM to establish superiority over ROUGE, METEOR, BLEU, and BERTScore.
• Perform ablations to test robustness to QA/QG model quality, domain shift, number of questions, and alternative similarity metrics.
• Show qualitative examples illustrating how QAGS identifies inconsistencies and provides interpretability by highlighting which tokens are problematic.

Experimental results

Research questions

• RQ1 Can QAGS reliably detect factual inconsistencies in abstractive summaries across CNN/DailyMail and XSUM?
• RQ2 How does QAGS correlate with human judgments of factuality compared to traditional metrics like ROUGE, BLEU, METEOR, and BERTScore?
• RQ3 Is QAGS robust to variations in QA/QG model quality, domain shifts, and the number of questions used?
• RQ4 Does QAGS provide interpretable insights into which parts of a summary are inconsistent?
• RQ5 How does QAGS perform in re-ranking or fact-checking contexts relative to NLI-based methods?

Key findings

• QAGS achieves substantially higher Pearson correlations with human judgments than ROUGE, METEOR, BLEU, and BERTScore on CNN/DM and XSUM (e.g., 54.53 vs. 17.72 for ROUGE-2 on CNN/DM).
• QAGS remains robust across QA/QG model quality and domain shifts, with correlations remaining strong even under weaker components.
• Increasing the number of questions from 5 to 20 yields notable gains in correlation, with diminishing returns beyond ~50 questions.
• QAGS outperforms NLI-based approaches (e.g., BERT NLI, ESIM, FactCC) on a sentence ranking task for factual consistency (72.1% vs. 64.1–70.0%).
• QAGS provides interpretable outputs through the generated questions and answers, indicating which tokens are inconsistent in the summary.