[Paper Review] Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task

Yang An, Kai Liu+3 more|arXiv (Cornell University)|Jun 10, 2018

Topic Modeling15 references3 citations

TL;DR

This paper proposes adapted versions of ROUGE and BLEU that incorporate opinion-aware and entity-aware bonus terms to improve evaluation of machine reading comprehension systems, especially for yes-no and entity-list questions. The adapted metrics achieve significantly higher correlation with human judgments—up to 0.570 Pearson correlation for ROUGE-L on overall scores—demonstrating improved alignment with human evaluation for these challenging question types.

ABSTRACT

Current evaluation metrics to question answering based machine reading comprehension (MRC) systems generally focus on the lexical overlap between the candidate and reference answers, such as ROUGE and BLEU. However, bias may appear when these metrics are used for specific question types, especially questions inquiring yes-no opinions and entity lists. In this paper, we make adaptations on the metrics to better correlate n-gram overlap with the human judgment for answers to these two question types. Statistical analysis proves the effectiveness of our approach. Our adaptations may provide positive guidance for the development of real-scene MRC systems.

Motivation & Objective

• Address the limitation of standard ROUGE and BLEU metrics in evaluating machine reading comprehension (MRC) systems, particularly for yes-no and entity-list questions.
• Identify that lexical overlap-based metrics may fail to capture semantic agreement in answers with opposite opinions or missing/misclassified entities.
• Develop a unified, automatic evaluation framework that enhances ROUGE and BLEU by incorporating domain-specific awareness of yes-no opinions and entity correctness.
• Improve alignment between automatic metrics and human judgment, especially for real-world MRC scenarios where such question types are prevalent.
• Provide a scalable, end-to-end solution that avoids manual annotation while improving evaluation reliability for diverse question types.

Proposed method

• Introduce a yes-no opinion bonus term that rewards agreement in binary opinion (yes/no) between candidate and reference answers, based on lexical overlap of opinion words.
• Introduce an entity bonus term that rewards correct listing of entities, with a focus on precision and recall of named entities in the answer.
• Modify ROUGE-L and BLEU-4 by integrating the opinion and entity bonus terms into their standard scoring formulas, adjusting the final score to reflect semantic agreement beyond n-gram overlap.
• Use a weighted combination of original metric scores and bonus terms, with hyperparameters α (for yes-no) and β (for entities) set to 2.0 and 1.0 respectively.
• Apply paired bootstrap resampling to test the statistical significance of performance improvements over baseline metrics.
• Validate the adapted metrics on the DuReader dataset using human-annotated judgments for both single-question and overall score evaluation levels.

Experimental results

Research questions

• RQ1 To what extent do standard ROUGE and BLEU metrics correlate with human judgment for yes-no and entity-list questions in MRC?
• RQ2 Can introducing opinion-aware and entity-aware bonus terms improve the correlation between automatic metrics and human evaluation?
• RQ3 How do the weights of the bonus terms (α and β) affect the performance of the adapted metrics?
• RQ4 Does the adapted ROUGE-L outperform both original ROUGE-L and adapted BLEU-4 in evaluating MRC systems across different question types?
• RQ5 Is the improvement of the adapted metrics statistically significant compared to the original metrics?

Key findings

• The adapted ROUGE-L achieves a Pearson correlation coefficient (PCC) of 0.570 with human judgment on overall scores, significantly higher than the original ROUGE-L (0.504) and adapted BLEU-4 (0.481) on single-question evaluation.
• On overall score evaluation, adapted ROUGE-L achieves a PCC of 0.792, outperforming original ROUGE-L (0.760) and adapted BLEU-4 (0.646), with a 4.2% relative improvement over the original.
• The adapted BLEU-4 shows a significant improvement on entity answers (PCC 0.686 vs. 0.668), but performs worse than original BLEU-4 on yes-no questions, indicating potential issues with BLEU’s decomposability.
• Statistical paired bootstrap tests confirm that the adapted metrics significantly outperform their original counterparts across all question types (yes-no, entity, overall) with p < 0.05 in all 6 tests.
• The impact of bonus weights is monotonic: increasing α (yes-no bonus) and β (entity bonus) consistently improves PCCs on respective question types, suggesting tunable and scalable design.
• The results demonstrate that incorporating semantic-aware bonus terms into standard metrics enhances their ability to reflect human judgment, especially for non-factoid, real-world MRC questions.