[Paper Review] SumQE: a BERT-based Summary Quality Estimation Model

Stratos Xenouleas, Prodromos Malakasiotis+2 more|arXiv (Cornell University)|Sep 2, 2019

Natural Language Processing Techniques22 references4 citations

TL;DR

This paper proposes Sum-QE, a BERT-based model for summarization quality estimation that predicts linguistic quality without human references. It achieves high correlation with human judgments on five criteria—grammaticality, non-redundancy, referential clarity, focus, and structure—outperforming baseline models, especially on fluency and coherence aspects.

ABSTRACT

We propose SumQE, a novel Quality Estimation model for summarization based on BERT. The model addresses linguistic quality aspects that are only indirectly captured by content-based approaches to summary evaluation, without involving comparison with human references. SumQE achieves very high correlations with human ratings, outperforming simpler models addressing these linguistic aspects. Predictions of the SumQE model can be used for system development, and to inform users of the quality of automatically produced summaries and other types of generated text.

Motivation & Objective

• To develop a reference-free quality estimation model for summarization that captures linguistic quality aspects beyond content preservation.
• To address the limitations of content-based metrics like ROUGE and Pyramid, which fail to assess fluency, coherence, and grammatical correctness.
• To leverage BERT's contextual representations to predict human-annotated linguistic quality scores on multiple datasets.
• To enable real-time quality assessment during system development and user-facing text generation.
• To extend the applicability of quality estimation to other text generation tasks beyond summarization.

Proposed method

• Fine-tune a pre-trained BERT model with a task-specific regression head to predict five linguistic quality scores (Q1–Q5) from human annotations.
• Use three NIST DUC shared task datasets (DUC-05, DUC-06, DUC-07) with human-rated summaries on a 1–5 scale for each quality criterion.
• Train the model end-to-end on summary text alone, without requiring reference summaries or post-editing data.
• Evaluate performance using Spearman’s rank correlation between predicted and human-annotated scores.
• Compare Sum-QE against baseline models including BERT, BiLSTM, and ROUGE-based methods to isolate linguistic quality prediction capability.
• Apply multi-task learning in some variants to jointly predict correlated quality aspects (e.g., Q4 and Q5), improving generalization.

Experimental results

Research questions

• RQ1 Can a BERT-based model accurately estimate linguistic quality of summaries without relying on human reference summaries?
• RQ2 How well does Sum-QE correlate with human judgments across distinct linguistic quality dimensions such as grammaticality, coherence, and non-redundancy?
• RQ3 Does the model outperform simpler baselines in predicting subtle linguistic qualities that content-based metrics miss?
• RQ4 Why does Sum-QE struggle with certain criteria like non-redundancy (Q2) in specific datasets, and what architectural factors influence this?
• RQ5 To what extent can Sum-QE be generalized to other domains or text generation tasks with minimal fine-tuning?

Key findings

• Sum-QE achieves very high Spearman correlation with human ratings across all five linguistic quality criteria on the DUC-05, DUC-06, and DUC-07 datasets.
• The model shows particularly strong performance on grammaticality (Q1), referential clarity (Q3), focus (Q4), and structure/coherence (Q5), with correlations above 0.8 in most cases.
• For non-redundancy (Q2), correlation is moderate (0.65) in DUC-05 due to high human score variance being low and scores clustered between 4 and 5.
• BERT-based models outperform BiLSTM and ROUGE-based baselines, especially on long-range dependencies critical for coherence and focus.
• Multi-task BERT variants show improved performance on highly correlated criteria like Q4 and Q5, suggesting shared representation learning helps.
• The model generalizes well to other domains with minimal fine-tuning, and can be applied to other text generation tasks such as sentence compression or NLG.