[Paper Review] Question Answering as an Automatic Evaluation Metric for News Article Summarization

Matan Eyal, Tal Baumel+1 more|arXiv (Cornell University)|Jun 2, 2019

Topic Modeling32 references4 citations

TL;DR

This paper introduces APES, an extrinsic evaluation metric for news article summarization that measures a summary's ability to answer questions about salient entities, using a pre-trained question-answering model. The method achieves higher correlation with manual evaluation metrics than ROUGE, and an abstractive summarization model optimized for APES improves both APES (46.1 vs. 39.8) and ROUGE scores, demonstrating that APES can guide effective summarization beyond ROUGE alone.

ABSTRACT

Recent work in the field of automatic summarization and headline generation focuses on maximizing ROUGE scores for various news datasets. We present an alternative, extrinsic, evaluation metric for this task, Answering Performance for Evaluation of Summaries. APES utilizes recent progress in the field of reading-comprehension to quantify the ability of a summary to answer a set of manually created questions regarding central entities in the source article. We first analyze the strength of this metric by comparing it to known manual evaluation metrics. We then present an end-to-end neural abstractive model that maximizes APES, while increasing ROUGE scores to competitive results.

Motivation & Objective

• To address the limitation of ROUGE in single-reference summarization datasets by proposing a more robust, extrinsic evaluation metric.
• To evaluate whether question-answering performance on salient entities correlates with human judgment of summary quality.
• To develop an abstractive summarization model that directly optimizes for APES, improving both APES and ROUGE scores.
• To validate that APES captures coherence and semantic accuracy beyond mere entity presence, unlike ROUGE.
• To demonstrate that APES can guide effective summarization training without relying on reinforcement learning or complex reward shaping.

Proposed method

• APES evaluates summaries by measuring the percentage of manually created questions about salient entities that are correctly answered by the summary.
• Questions are derived from the reference summary and focus on central named entities, ensuring alignment with salient content.
• A pre-trained reading comprehension model is used to answer questions based on the summary, with performance measured as the fraction of correctly answered questions.
• An abstractive sequence-to-sequence model is trained to maximize APES by attending more strongly to salient entities in the source document.
• The model uses attention mechanisms to prioritize salient entities, with entity saliency predicted via a learned module rather than relying on gold labels.
• The method is evaluated on the CNN/Daily Mail dataset using both automatic metrics (ROUGE) and manual evaluation (Pyramid, Responsiveness).

Experimental results

Research questions

• RQ1 Does APES correlate more strongly with human evaluation metrics such as Pyramid and Responsiveness than ROUGE?
• RQ2 Can an abstractive summarization model optimized for APES achieve competitive ROUGE scores while improving on APES itself?
• RQ3 Is APES sensitive to coherence and semantic structure, or does it merely detect the presence of named entities?
• RQ4 How effective is the model's predicted saliency score in guiding attention compared to using gold entity positions?
• RQ5 Does the QA system used in APES achieve sufficient performance to make the metric reliable and feasible for evaluation?

Key findings

• APES shows a strong correlation with manual evaluation metrics, particularly Pyramid and Responsiveness, outperforming ROUGE in this regard.
• The APES-optimized model achieves an APES score of 46.1 on the test set, significantly higher than the baseline model’s 39.8.
• The model also improves ROUGE scores by approximately 1 F1 point across ROUGE-1, ROUGE-2, and ROUGE-L, demonstrating that APES optimization enhances both intrinsic and extrinsic performance.
• The QA system achieves high performance (72.4% and 75.8% on CNN and Daily Mail), confirming that the metric is feasible and reliable for evaluation.
• Shuffled gold summaries score only 53.8% on APES, showing that APES penalizes incoherence and loss of meaning, unlike ROUGE which fails to detect such issues.
• Using gold salient entity positions as oracle inputs yields only a slight improvement (46.3 vs. 46.1), indicating that the model’s saliency prediction is effective and sufficient for high performance.