[Paper Review] BLEU is Not Suitable for the Evaluation of Text Simplification

Elior Sulem, Omri Abend+1 more|arXiv (Cornell University)|Oct 14, 2018

Text Readability and Simplification21 references4 citations

TL;DR

This paper demonstrates that BLEU is unsuitable for evaluating text simplification, particularly sentence splitting, due to its negative correlation with simplicity and poor alignment with human judgments. Using a new human-annotated corpus (HSplit), the authors show BLEU fails to capture grammaticality, meaning preservation, and especially simplicity—often penalizing simpler, split sentences, rendering it misleading for structural simplification tasks.

ABSTRACT

BLEU is widely considered to be an informative metric for text-to-text generation, including Text Simplification (TS). TS includes both lexical and structural aspects. In this paper we show that BLEU is not suitable for the evaluation of sentence splitting, the major structural simplification operation. We manually compiled a sentence splitting gold standard corpus containing multiple structural paraphrases, and performed a correlation analysis with human judgments. We find low or no correlation between BLEU and the grammaticality and meaning preservation parameters where sentence splitting is involved. Moreover, BLEU often negatively correlates with simplicity, essentially penalizing simpler sentences.

Motivation & Objective

• To evaluate the suitability of BLEU for text simplification (TS), particularly sentence splitting, a core structural operation.
• To address the lack of reliable automatic evaluation metrics for structural simplification, especially when sentence splitting is involved.
• To investigate whether BLEU can be adapted to assess sentence splitting using a reference set focused on splitting.
• To compare BLEU’s performance against simpler baselines like -LD SC in capturing human judgment dimensions such as simplicity, grammaticality, and meaning preservation.

Proposed method

• Constructed a new human-annotated gold standard corpus, HSplit, containing multiple structural paraphrases of sentences with sentence splitting.
• Performed system-level and sentence-level Spearman correlation analyses between automatic metrics (BLEU, iBLEU, SARI, -LD SC) and human judgments on grammaticality (G), meaning preservation (M), simplicity (S), and structural simplicity (StS).
• Evaluated BLEU using two reference settings: a standard reference set emphasizing lexical operations (Xu et al., 2016) and the HSplit corpus focused on sentence splitting.
• Computed correlation coefficients and p-values to assess the reliability of BLEU and baselines in predicting human judgments.
• Compared BLEU’s performance against -LD SC, a simple similarity measure to the source, to test if BLEU acts merely as a conservatism proxy.
• Analyzed the impact of sentence splitting on BLEU scores by comparing results across systems with and without splitting operations.

Experimental results

Research questions

• RQ1 Does BLEU correlate with human judgments of simplicity in text simplification, especially when sentence splitting is involved?
• RQ2 Can BLEU reliably measure grammaticality and meaning preservation in text simplification outputs, particularly when sentence splitting occurs?
• RQ3 Is BLEU’s performance improved when evaluated against a reference set specifically designed for sentence splitting, such as HSplit?
• RQ4 Does BLEU act primarily as a measure of conservatism (similarity to source) rather than a true indicator of simplification quality?
• RQ5 How does BLEU compare to simpler baselines like -LD SC in predicting human judgments of simplification quality?

Key findings

• BLEU exhibits a negative correlation with simplicity (S) and structural simplicity (StS), with system-level Spearman correlations of -0.70 (p=0.06) and -0.60 (p=0.1) respectively when using HSplit as the reference set.
• BLEU shows low or no correlation with meaning preservation (M) and grammaticality (G) when sentence splitting is involved, with correlations of 0.11 (p=0.4) and 0.57 (p=0.1) respectively in the HSplit setting.
• Even in the standard reference setting, BLEU negatively correlates with simplicity (S) and structural simplicity (StS), with correlations of 0.17 (p=3×10⁻⁴) and 0.17 (p=3×10⁻⁴), indicating poor alignment with human judgments.
• The correlation between BLEU and -LD SC is high (0.86 for BLEU-1ref, p=7×10⁻³) without splits and 0.52 (p=0.04) with splits, suggesting BLEU may primarily measure conservatism rather than simplification quality.
• SARI shows only weak positive correlation with simplicity (0.26 at system level) and no significant correlation with structural simplicity, indicating its limited sensitivity to structural changes.
• iBLEU performs comparably to BLEU but with slightly lower correlations, confirming that the issue is not specific to BLEU’s variant but inherent to the metric’s design for structural simplification tasks.