[Paper Review] QuestEval: Summarization Asks for Fact-based Evaluation

Thomas Scialom, Paul-Alexis Dray+5 more|arXiv (Cornell University)|Mar 23, 2021

Topic ModelingComputer Science35 references25 citations

TL;DR

QuestEval is a reference-free summarization evaluation metric that unifies precision and recall QA-based signals via a QA model and a question-weighted generator, achieving higher correlation with human judgments without needing gold references.

ABSTRACT

Summarization evaluation remains an open research problem: current metrics such as ROUGE are known to be limited and to correlate poorly with human judgments. To alleviate this issue, recent work has proposed evaluation metrics which rely on question answering models to assess whether a summary contains all the relevant information in its source document. Though promising, the proposed approaches have so far failed to correlate better than ROUGE with human judgments. In this paper, we extend previous approaches and propose a unified framework, named QuestEval. In contrast to established metrics such as ROUGE or BERTScore, QuestEval does not require any ground-truth reference. Nonetheless, QuestEval substantially improves the correlation with human judgments over four evaluation dimensions (consistency, coherence, fluency, and relevance), as shown in the extensive experiments we report.

Motivation & Objective

• Address the limitations of n-gram metrics like ROUGE for summarization evaluation.
• Develop a ground-truth-free metric that better correlates with human judgments across multiple dimensions.
• Unify precision and recall QA-based evaluations and introduce a learned question weighting to capture saliency.
• Demonstrate strong correlation with human judgments on CNN/Daily Mail and XSUM datasets, including factual consistency.
• Provide an explainable evaluation framework that does not require reference summaries.

Proposed method

• Propose QuestEval, a unified QA-based framework combining precision (QG/QA with source) and recall (weighted QG/QA with source) signals.
• Use a pretrained T5-based QA model to answer generated questions from source and summary, including an unanswerable token epsilon.
• Generate questions from source or summary conditioned on answer candidates using a T5-based QG model, retaining only questions the QA model answers correctly.
• Introduce a learned question weighter W to prioritize salient questions for recall, trained on whether the summary contains answers.
• Compute precision as the F1 overlap between QA-predicted answers on the source and ground-truth answers, and recall via a weighted, answerability-aware measure.
• Unify precision and recall via their harmonic mean (F1) to produce the QuestEval score without requiring references.

Experimental results

Research questions

• RQ1 Can a reference-free QA-based evaluation better capture factual consistency and information saliency in summaries than reference-based metrics?
• RQ2 Does unifying precision and recall QA signals and introducing a learned question weighting improve alignment with human judgments across consistency, coherence, fluency, and relevance?
• RQ3 How does QuestEval perform on major summarization datasets (CNN/Daily Mail and XSUM) compared to ROUGE, BLEU, METEOR, BERTScore, and existing QA-based metrics?
• RQ4 Is QuestEval robust to varying numbers of references and capable of evaluating systems when no gold reference is available?

Key findings

• QuestEval markedly improves correlation with human judgments across consistency, coherence, fluency, and relevance compared to baseline metrics.
• The unified precision-recall approach with a learned question weighter yields higher average correlations than SummaQA and QAGS; learned weighting improves Relevance.
• QuestEval remains reference-free and maintains strong performance even when only a single reference is available, and it outperforms reference-based metrics when references are scarce.
• Incorporating negative sampling for QA training enhances performance, particularly in detecting unanswerable questions and reducing hallucinations.
• The framework provides explainability by showing which questions are important and answered, highlighting factual gaps and inconsistencies in summaries.