[Paper Review] Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning

Hanlu Wu, Tengfei Ma+3 more|arXiv (Cornell University)|Oct 5, 2020

Topic Modeling28 references4 citations

TL;DR

This paper proposes a reference-free, unsupervised summary quality evaluation method using contrastive learning to assess both linguistic and semantic qualities of summaries without human references. By constructing task-specific negative samples and training a BERT-based evaluator via ranking loss, the method achieves state-of-the-art correlation with human judgments on Newsroom and CNN/Daily Mail, demonstrating strong generalization across datasets.

ABSTRACT

Evaluation of a document summarization system has been a critical factor to impact the success of the summarization task. Previous approaches, such as ROUGE, mainly consider the informativeness of the assessed summary and require human-generated references for each test summary. In this work, we propose to evaluate the summary qualities without reference summaries by unsupervised contrastive learning. Specifically, we design a new metric which covers both linguistic qualities and semantic informativeness based on BERT. To learn the metric, for each summary, we construct different types of negative samples with respect to different aspects of the summary qualities, and train our model with a ranking loss. Experiments on Newsroom and CNN/Daily Mail demonstrate that our new evaluation method outperforms other metrics even without reference summaries. Furthermore, we show that our method is general and transferable across datasets.

Motivation & Objective

• To develop an automatic, reference-free evaluation metric for text summarization that does not rely on human-annotated references.
• To improve evaluation by jointly modeling linguistic quality (e.g., fluency, grammar) and semantic informativeness (e.g., relevance, redundancy).
• To enable unsupervised training of a summary evaluator using contrastive learning with no human ratings or reference summaries.
• To ensure the method generalizes across different summarization datasets without retraining.

Proposed method

• Design a BERT-based evaluator that scores summaries based on both linguistic and semantic qualities.
• Construct multiple types of negative samples by perturbing summaries along linguistic and semantic dimensions (e.g., word replacement, sentence shuffling, sentence deletion).
• Train the evaluator using a contrastive learning framework with a ranking loss that maximizes the score difference between original and negative samples.
• Use document-level BERT embeddings as a source of semantic supervision, replacing reliance on reference summaries.
• Optimize the model end-to-end using contrastive loss to align high-quality summaries closer to the positive anchor than to any negative sample.
• Enable cross-dataset transferability by training on one dataset and testing on another without fine-tuning.

Experimental results

Research questions

• RQ1 Can a reference-free, unsupervised method achieve high correlation with human judgments in summarization evaluation?
• RQ2 Can contrastive learning effectively train a summary evaluator without human-annotated ratings or reference summaries?
• RQ3 Does jointly modeling linguistic and semantic qualities improve evaluation performance compared to methods focusing only on semantic similarity?
• RQ4 Can the trained evaluator generalize across different summarization datasets without retraining?

Key findings

• The proposed method, LS_Score, achieves the highest Spearman correlation with human judgments on both Newsroom (0.6563 overall) and CNN/Daily Mail (0.3342 overall), outperforming ROUGE, BERTScore-R, MoverScore, and BERT+Cos+Doc.
• On Newsroom, LS_Score achieves 0.7163 correlation for informativeness and 0.6563 for overall quality, significantly outperforming BERTScore-R (0.2972 and 0.2787).
• On CNN/Daily Mail, LS_Score achieves 0.5933 for fluency and 0.2875 for redundancy, outperforming BERT+Linear (0.4511 and 0.1664) and BERTScore-R (0.2227 and 0.2780).
• The ablation study confirms that the contrastive learning framework with the proposed evaluator is superior to a standard BERT+Linear baseline, even when both use the same contrastive setup.
• Cross-dataset transfer experiments show LS_Score_cross maintains strong performance (e.g., 0.6271 on Newsroom and 0.2874 on CNN/Daily Mail), proving the method's generalization capability.
• The model trained on CNN/Daily Mail generalizes well to Newsroom, and vice versa, demonstrating robust transferability across datasets.