[Paper Review] Traditional Readability Formulas Compared for English

Bruce W. Lee, Jason Hyung-Jong Lee|arXiv (Cornell University)|Jan 8, 2023

Text Readability and Simplification4 citations

TL;DR

This paper introduces a recalibrated set of traditional English readability formulas—Flesch-Kincaid, Fog, SMOG, Coleman-Liau, and Automated Readability Index—optimized for modern U.S. educational texts, and proposes a new formula, NERF, based on lexico-semantic and syntactic features. Using the Common Core Appendix B dataset, the authors improve prediction accuracy and release a Python-based tool for broad NLP application.

ABSTRACT

Traditional English readability formulas, or equations, were largely developed in the 20th century. Nonetheless, many researchers still rely on them for various NLP applications. This phenomenon is presumably due to the convenience and straightforwardness of readability formulas. In this work, we contribute to the NLP community by 1. introducing New English Readability Formula (NERF), 2. recalibrating the coefficients of old readability formulas (Flesch-Kincaid Grade Level, Fog Index, SMOG Index, Coleman-Liau Index, and Automated Readability Index), 3. evaluating the readability formulas, for use in text simplification studies and medical texts, and 4. developing a Python-based program for the wide application to various NLP projects.

Motivation & Objective

• Address the persistent use of outdated readability formulas in NLP despite advances in ML-based models.
• Improve the accuracy of traditional readability formulas by recalibrating their coefficients using modern U.S. curriculum-aligned texts.
• Develop a new, generalized readability formula (NERF) grounded in lexico-semantic and syntactic features for broader linguistic coverage.
• Evaluate the performance of six readability formulas (five recalibrated + NERF) across diverse datasets representing different reading levels and audiences.
• Provide a fast, open-source Python tool for easy integration into NLP pipelines for readability assessment.

Proposed method

• Recalibrated five traditional readability formulas using the Common Core Appendix B (CCB) corpus as the primary training dataset.
• Identified key linguistic features from lexico-semantic (e.g., word familiarity via SubtlexUS frequency) and syntactic (e.g., part-of-speech ratios, phrase frequency) branches using cross-dataset correlation analysis.
• Proposed NERF as a new formula combining top-performing features from multiple datasets, optimized for generalizability and accuracy.
• Applied linear regression models to derive new coefficients for each formula using CCB’s grade-level annotations as ground truth.
• Evaluated formula performance across five benchmark datasets: CCB, WeeBit, Cambridge English, Korean ELT Curriculum, and OneStopEnglish.
• Developed a Python-based software package (<Anonymous>) for efficient, scalable readability assessment in NLP applications.

Experimental results

Research questions

• RQ1 How can traditional readability formulas be recalibrated to improve accuracy on modern U.S. educational texts?
• RQ2 Which linguistic features—lexico-semantic or syntactic—contribute most significantly to improved readability prediction across diverse text types?
• RQ3 How does the performance of the newly proposed NERF formula compare to existing traditional formulas on standardized datasets?
• RQ4 To what extent do recalibrated formulas generalize across different reading levels and educational curricula?
• RQ5 Can a unified, open-source Python tool effectively support the integration of readability assessment into diverse NLP workflows?

Key findings

• The recalibrated versions of Flesch-Kincaid, Fog, SMOG, Coleman-Liau, and Automated Readability Index formulas showed improved prediction accuracy on the Common Core Appendix B dataset compared to original coefficients.
• NERF, based on features like word familiarity (SubtlexUS frequency) and syntactic ratios (e.g., adverb-to-noun ratio), achieved superior generalization across multiple datasets.
• The feature 'at_SbFrL_C' (SubtlexUS FREQlow per word) showed the highest correlation (r = 0.176) with CCB grade levels, indicating strong predictive power for word familiarity.
• Syntactic features such as 'ra_AvNoT_C' (adverb-to-noun POS ratio) and 'LoCoDPU_S' (local coherence distance) demonstrated consistent performance across multiple datasets.
• The Python-based tool <Anonymous> enables efficient, scalable readability assessment with minimal computational overhead, supporting integration into NLP pipelines.
• NERF outperformed all traditional formulas in cross-dataset evaluation, particularly on lower- and mid-level education texts, suggesting broader linguistic coverage and robustness.