[Paper Review] ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Tasks

Kavita Ganesan|arXiv (Cornell University)|Mar 5, 2018

Natural Language Processing Techniques3 references105 citations

TL;DR

The paper introduces ROUGE 2.0, extending ROUGE with synonym-aware and topic-focused evaluation measures (ROUGE-N+Synonyms, ROUGE-Topic, ROUGE-Topic+Synonyms, ROUGE-TopicUniq, ROUGE-TopicUniq+Synonyms) and provides a Java implementation for improved, semantically aware summary evaluation.

ABSTRACT

Evaluation of summarization tasks is extremely crucial to determining the quality of machine generated summaries. Over the last decade, ROUGE has become the standard automatic evaluation measure for evaluating summarization tasks. While ROUGE has been shown to be effective in capturing n-gram overlap between system and human composed summaries, there are several limitations with the existing ROUGE measures in terms of capturing synonymous concepts and coverage of topics. Thus, often times ROUGE scores do not reflect the true quality of summaries and prevents multi-faceted evaluation of summaries (i.e. by topics, by overall content coverage and etc). In this paper, we introduce ROUGE 2.0, which has several updated measures of ROUGE: ROUGE-N+Synonyms, ROUGE-Topic, ROUGE-Topic+Synonyms, ROUGE-TopicUniq and ROUGE-TopicUniq+Synonyms; all of which are improvements over the core ROUGE measures.

Motivation & Objective

• Identify limitations of the original ROUGE measures in capturing synonyms, content coverage, and topic-level evaluation.
• Propose and define updated ROUGE measures that address semantic overlap and topic/subset coverage.
• Provide a Java-based implementation of ROUGE 2.0 and guidance for using domain-specific synonym dictionaries.

Proposed method

• Introduce ROUGE-{N|Topic|TopicUniq}+Synonyms to capture semantic overlap via a synonym dictionary.
• Define ROUGE-Topic and ROUGE-TopicUniq to evaluate topic or subset coverage using POS-based tokens.
• Provide formal definitions for ROUGE-Topic and ROUGE-TopicUniq with recall and precision formulations.
• Demonstrate how synonyms and topic-based measures affect recall, precision, and F-scores on example summaries.
• Describe the Java implementation and notes on linking to domain/language-specific synonym resources.

Experimental results

Research questions

• RQ1 Can ROUGE scores be misleading when synonyms are present or when only a subset of content topics is evaluated?
• RQ2 Do synonym-aware and topic-focused measures provide a more accurate reflection of summary quality across different content dimensions?
• RQ3 How do ROUGE-Topic and ROUGE-TopicUniq variants compare to standard ROUGE in measuring content coverage and conciseness?

Key findings

• ROUGE-1 + StopWordRemoval + Synonyms yields perfect recall (1.000) in the illustrated example, highlighting semantic overlap improvements.
• ROUGE-TopicNN|JJ and ROUGE-TopicUniqNN|JJ show improved topic coverage measurement when using synonyms, with higher F-scores in the example.
• ROUGE-TopicUniq variants reduce repetition and better reflect concise topic coverage compared to non-uniq topic measures.
• The ROUGE 2.0 package is implemented in Java for platform flexibility, with a modular synonym dictionary system and POS-based topic evaluation.
• Table-based demonstration (Example 1.1) shows how synonyms and unique-topic measures alter recall/precision/F-scores across variants.
• ROUGE-2.0 provides guidance for integrating domain-specific synonym dictionaries and multiple POS tag configurations.