[Paper Review] How to Find Strong Summary Coherence Measures? A Toolbox and a Comparative Study for Summary Coherence Measure Evaluation

Julius Steen, Katja Markert|arXiv (Cornell University)|Sep 14, 2022

Topic Modeling4 citations

TL;DR

This paper introduces a comprehensive toolbox and large-scale evaluation framework for summary coherence measures (CMs), proposing intra-system correlation and bias matrices to detect system-level confounders and model biases. It finds that while large language models fine-tuned on the shuffle task show strong performance, most existing CMs fail under stricter evaluation, especially when generalizing across diverse summary lengths and content, highlighting the need for classification-based training over pairwise ranking.

ABSTRACT

Automatically evaluating the coherence of summaries is of great significance both to enable cost-efficient summarizer evaluation and as a tool for improving coherence by selecting high-scoring candidate summaries. While many different approaches have been suggested to model summary coherence, they are often evaluated using disparate datasets and metrics. This makes it difficult to understand their relative performance and identify ways forward towards better summary coherence modelling. In this work, we conduct a large-scale investigation of various methods for summary coherence modelling on an even playing field. Additionally, we introduce two novel analysis measures, intra-system correlation and bias matrices, that help identify biases in coherence measures and provide robustness against system-level confounders. While none of the currently available automatic coherence measures are able to assign reliable coherence scores to system summaries across all evaluation metrics, large-scale language models fine-tuned on self-supervised tasks show promising results, as long as fine-tuning takes into account that they need to generalize across different summary lengths.

Motivation & Objective

• To address the lack of standardized evaluation for summary coherence measures (CMs), which are often assessed on disparate datasets and metrics, making performance comparisons unreliable.
• To identify and mitigate system-level confounders in CMs that exploit performance gaps between summarizers rather than true coherence modeling.
• To introduce new evaluation metrics—specifically intra-system correlation and bias matrices—that improve robustness and detect model biases.
• To evaluate a wide range of CMs on the recent SummEval dataset under consistent, rigorous conditions to assess generalizability and performance across diverse summary lengths and content.
• To guide future research by identifying limitations of current approaches and recommending improvements, especially in self-supervised training and model architecture.

Proposed method

• Propose intra-system correlation as a new evaluation metric that measures coherence score consistency within summaries generated by the same summarizer, reducing vulnerability to system-level confounders.
• Introduce bias matrices as a diagnostic tool to visualize and detect systematic biases of CMs toward specific summarizers or model types.
• Conduct a large-scale comparative study of 14 CMs across multiple evaluation regimes, including system-level correlation, pairwise ranking, and intra-system correlation.
• Fine-tune large language models (LLMs) on the sentence shuffle task using a binary classification objective (CCL), contrasting with standard pairwise ranking losses.
• Use modified shuffle tests to evaluate CMs on documents of varying lengths and content, simulating real-world diversity in summary generation.
• Apply the Brown Coherence Toolkit to quantify entity overlap across datasets, assessing the feasibility of entity-based coherence modeling.

Experimental results

Research questions

• RQ1 Can existing summary coherence measures reliably evaluate coherence across diverse summarizer outputs, or do they exploit system-level confounders?
• RQ2 How do different evaluation metrics—system-level correlation, pairwise ranking, and intra-system correlation—compare in revealing the true coherence modeling ability of CMs?
• RQ3 To what extent are coherence measures biased toward specific summarizers, and can bias matrices effectively detect such biases?
• RQ4 Why do CMs that perform well on the shuffle task fail on real-world summarization datasets like SummEval, especially when summary lengths vary?
• RQ5 Does training coherence models with a global classification objective (e.g., CCL) improve generalization across different summary lengths and content compared to pairwise ranking?

Key findings

• None of the evaluated coherence measures achieve reliable performance across all evaluation metrics, particularly under intra-system correlation, indicating significant limitations in generalization.
• Even top-performing CMs exhibit strong biases toward specific summarizers, as revealed by bias matrices, raising concerns about their reliability and fairness.
• Entity-based coherence models (e.g., EEG, EGR, NEG) perform poorly due to minimal lexical entity overlap in CNN/DM and SummEval summaries, limiting their applicability.
• Models trained with pairwise ranking losses (e.g., UNF, GRA) fail to generalize to summaries of different lengths, while the CCL model with a classification objective maintains stable performance across length variations.
• Fine-tuned large language models on the shuffle task, especially when trained as a binary classifier, show the most promising results, suggesting a viable path forward for future coherence modeling.
• The study demonstrates that system-level correlation is insufficient for evaluating coherence models, as it is vulnerable to confounding factors and does not reflect true coherence understanding.