[Paper Review] A Semantically Motivated Approach to Compute ROUGE Scores

Elaheh ShafieiBavani, Mohammad Ebrahimi+2 more|arXiv (Cornell University)|Oct 20, 2017

Topic Modeling11 references3 citations

TL;DR

This paper proposes GRouge, a semantically enhanced variant of the ROUGE metric that integrates lexical and semantic similarity using Personalized PageRank over WordNet to improve correlation with human judgments. By combining n-gram matching with sense-level semantic similarity, GRouge significantly outperforms standard ROUGE on TAC AESOP datasets, especially for abstractive summaries with paraphrasing.

ABSTRACT

ROUGE is one of the first and most widely used evaluation metrics for text summarization. However, its assessment merely relies on surface similarities between peer and model summaries. Consequently, ROUGE is unable to fairly evaluate abstractive summaries including lexical variations and paraphrasing. Exploring the effectiveness of lexical resource-based models to address this issue, we adopt a graph-based algorithm into ROUGE to capture the semantic similarities between peer and model summaries. Our semantically motivated approach computes ROUGE scores based on both lexical and semantic similarities. Experiment results over TAC AESOP datasets indicate that exploiting the lexico-semantic similarity of the words used in summaries would significantly help ROUGE correlate better with human judgments.

Motivation & Objective

• To address ROUGE's overreliance on surface-level lexical overlap, which hinders fair evaluation of abstractive summaries with paraphrasing.
• To improve ROUGE's correlation with human judgments by incorporating semantic similarity between words and phrases.
• To develop a hybrid metric that balances lexical matching and semantic understanding for more robust automatic summarization evaluation.
• To enable ROUGE to better evaluate summaries involving lexical variation and paraphrasing, particularly in abstractive summarization.

Proposed method

• Apply Personalized PageRank (PPR) to WordNet 3.0 to compute sense-level semantic similarity distributions for words in summaries.
• Disambiguate word senses before computing similarity to ensure accurate semantic matching.
• Integrate semantic similarity scores with standard ROUGE n-gram match counts using a weighted combination controlled by a scaling factor β.
• Use a graph-based approach to compute lexico-semantic similarity between peer and model summaries at the n-gram level.
• Optimize the scaling factor β to balance lexical and semantic contributions, with β=0.5 yielding best performance.
• Apply pairwise Williams significance tests to validate statistical improvements over baseline ROUGE variants.

Experimental results

Research questions

• RQ1 Can integrating semantic similarity into ROUGE improve its correlation with human judgments on summarization quality?
• RQ2 How does the balance between lexical and semantic similarity (controlled by β) affect the performance of the enhanced metric?
• RQ3 Does the proposed GRouge method significantly outperform standard ROUGE variants in evaluating abstractive summaries with paraphrasing?
• RQ4 To what extent does the use of sense-level semantic analysis reduce the bias of ROUGE toward lexical overlap?
• RQ5 Can GRouge effectively evaluate summaries where lexical forms differ but meaning is preserved?

Key findings

• GRouge variants significantly outperform standard ROUGE and ROUGE-WE across all three human judgment metrics (Pyramid, Responsiveness, Readability) on TAC 2010 and 2011 AESOP datasets.
• All improvements in correlation from GRouge over ROUGE were statistically significant (p < 0.05) according to the pairwise Williams significance test.
• The optimal scaling factor β was found to be 0.5, balancing lexical and semantic contributions; performance degraded when β approached 0 or 1.
• GRouge-2 showed the strongest improvement, particularly in Spearman and Kendall rank correlations, indicating better ranking consistency with human judgments.
• The approach demonstrated robustness in handling paraphrased content, such as 'It is raining heavily' vs. 'It is pouring', which ROUGE fails to recognize as semantically equivalent.
• The results suggest GRouge is well-suited for evaluating abstractive summaries and has potential for application in text simplification evaluation.