[Paper Review] Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors

Liyan Tang, Tanya Goyal+7 more|arXiv (Cornell University)|May 25, 2022

Topic Modeling5 citations

TL;DR

This paper introduces AggreFact, a unified benchmark that aggregates factual error annotations from nine existing datasets, stratified by summarization model (FtSota, EXformer, Old). It reveals that state-of-the-art factuality metrics perform significantly worse on summaries from modern models (FtSota) than on older ones, challenging the validity of prior performance claims and urging evaluation on current models and error types for reliable metric assessment.

ABSTRACT

The propensity of abstractive summarization models to make factual errors has been studied extensively, including design of metrics to detect factual errors and annotation of errors in current systems' outputs. However, the ever-evolving nature of summarization systems, metrics, and annotated benchmarks makes factuality evaluation a moving target, and drawing clear comparisons among metrics has become increasingly difficult. In this work, we aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. We compare performance of state-of-the-art factuality metrics, including recent ChatGPT-based metrics, on this stratified benchmark and show that their performance varies significantly across different types of summarization models. Critically, our analysis shows that much of the recent improvement in the factuality detection space has been on summaries from older (pre-Transformer) models instead of more relevant recent summarization models. We further perform a finer-grained analysis per error-type and find similar performance variance across error types for different factuality metrics. Our results show that no one metric is superior in all settings or for all error types, and we provide recommendations for best practices given these insights.

Motivation & Objective

• To address the lack of consistent, up-to-date benchmarks for evaluating factual error detection in abstractive summarization.
• To identify why prior factuality metric evaluations may be misleading due to reliance on outdated models and datasets.
• To provide a stratified benchmark that separates errors by underlying summarization model and error type for finer-grained analysis.
• To guide best practices in selecting and developing factuality metrics by evaluating their performance across different models and error types.
• To advocate for ongoing, domain-extended benchmarks that evolve with new summarization models, especially LLM-generated summaries.

Proposed method

• Aggregated factuality error annotations from nine existing datasets into a single benchmark, AggreFact, with consistent stratification by summarization model (FtSota, EXformer, Old).
• Stratified the benchmark based on the development timeline of the underlying summarization models to isolate performance differences across model eras.
• Unified fine-grained error annotations from multiple datasets into a common taxonomy to enable cross-dataset comparison of error types.
• Evaluated nine state-of-the-art factuality metrics—including recent LLM-based ones—on the stratified benchmark to assess performance variance across models and error types.
• Performed detailed error-type-level analysis to compare metric capabilities across intrinsic/extrinsic, entity, event, and noun phrase-level hallucinations.
• Provided recommendations for best practices in metric selection and benchmarking based on empirical findings.

Experimental results

Research questions

• RQ1 Do factuality metrics perform consistently across different types of summarization models, particularly between modern (FtSota) and older (Old) models?
• RQ2 To what extent do recent improvements in factuality metrics stem from detecting errors in outdated models rather than current state-of-the-art systems?
• RQ3 How do different factuality metrics compare in identifying specific error types (e.g., intrinsic vs. extrinsic hallucinations) across datasets like CNN/DM and XSum?
• RQ4 Can a single factuality metric reliably detect all types of factual errors across diverse summarization models and datasets?
• RQ5 What are the best practices for evaluating and selecting factuality metrics in light of performance variance across models and error types?

Key findings

• Factuality metrics show significantly lower performance on summaries generated by state-of-the-art (FtSota) models compared to older models, indicating that recent metric gains are often not relevant to current systems.
• Most reported improvements in factuality metrics are achieved on the 'Old' and 'EXformer' subsets of the benchmark, not on the more relevant FtSota models, calling into question the validity of prior benchmarking practices.
• No single factuality metric outperforms all others across all model types or error types, highlighting the need for context-aware metric selection.
• Metrics claiming SOTA performance still fail to identify certain error types equally well across datasets like CNN/DM and XSum, revealing persistent gaps in coverage.
• The distribution of error types varies significantly across datasets and over time, even for the same summarization model, complicating cross-dataset comparisons.
• There is a critical need for updated, living benchmarks that include LLM-generated summaries and are regularly extended to reflect new model capabilities.