[Paper Review] A Readable Read: Automatic Assessment of Language Learning Materials based on Linguistic Complexity

Ildikó Pilán, Sowmya Vajjala+1 more|arXiv (Cornell University)|Mar 29, 2016

Text Readability and Simplification7 citations

TL;DR

This paper proposes a supervised machine learning model to automatically assess the linguistic complexity of Swedish second language learning materials on a 5-point difficulty scale, using a range of linguistic features. The model achieves 81.3% accuracy at the document level and 63.4% at the sentence level, with a 7% accuracy gain when combining multiple feature types, outperforming prior work in non-English languages.

ABSTRACT

Corpora and web texts can become a rich language learning resource if we have a means of assessing whether they are linguistically appropriate for learners at a given proficiency level. In this paper, we aim at addressing this issue by presenting the first approach for predicting linguistic complexity for Swedish second language learning material on a 5-point scale. After showing that the traditional Swedish readability measure, Lasbarhetsindex (LIX), is not suitable for this task, we propose a supervised machine learning model, based on a range of linguistic features, that can reliably classify texts according to their difficulty level. Our model obtained an accuracy of 81.3% and an F-score of 0.8, which is comparable to the state of the art in English and is considerably higher than previously reported results for other languages. We further studied the utility of our features with single sentences instead of full texts since sentences are a common linguistic unit in language learning exercises. We trained a separate model on sentence-level data with five classes, which yielded 63.4% accuracy. Although this is lower than the document level performance, we achieved an adjacent accuracy of 92%. Furthermore, we found that using a combination of different features, compared to using lexical features alone, resulted in 7% improvement in classification accuracy at the sentence level, whereas at the document level, lexical features were more dominant. Our models are intended for use in a freely accessible web-based language learning platform for the automatic generation of exercises.

Motivation & Objective

• To develop a reliable method for assessing linguistic complexity in Swedish second language learning materials.
• To address the limitation of the traditional LIX readability measure for language learning contexts.
• To create a model that classifies texts into five difficulty levels suitable for language learning platforms.
• To evaluate the performance of linguistic features at both document and sentence levels for complexity prediction.
• To support the automatic generation of language learning exercises through a freely accessible web-based platform.

Proposed method

• A supervised machine learning model is trained using a diverse set of linguistic features, including lexical, syntactic, and structural indicators of text complexity.
• The model is trained and evaluated on a dataset of Swedish second language texts annotated with difficulty levels on a 5-point scale.
• Feature engineering includes word frequency, sentence length, clause complexity, and lexical diversity metrics to capture linguistic difficulty.
• The model is evaluated at both document level (full texts) and sentence level (individual sentences) to assess scalability across linguistic units.
• A comparison between models using only lexical features and those combining multiple feature types is conducted to assess feature utility.
• The final model is designed for integration into a web-based language learning platform to enable automatic exercise generation.

Experimental results

Research questions

• RQ1 Can a machine learning model reliably predict the linguistic complexity of Swedish second language texts on a 5-point difficulty scale?
• RQ2 How does the performance of the model differ when classifying full texts versus individual sentences?
• RQ3 To what extent do different types of linguistic features (e.g., lexical vs. syntactic) contribute to classification accuracy?
• RQ4 Is combining multiple feature types more effective than relying on lexical features alone for complexity prediction?
• RQ5 How does the proposed model compare to existing readability measures like LIX in the context of second language learning materials?

Key findings

• The proposed machine learning model achieved an accuracy of 81.3% in classifying Swedish second language texts into five difficulty levels at the document level.
• The model's F-score was 0.8, indicating strong performance that is comparable to state-of-the-art results in English and superior to previous results in other languages.
• At the sentence level, the model achieved 63.4% accuracy, with an adjacent accuracy of 92%, suggesting strong performance for immediate context prediction.
• Combining multiple feature types improved classification accuracy by 7% at the sentence level, whereas lexical features were most influential at the document level.
• The traditional LIX readability measure was found unsuitable for classifying second language learning materials, highlighting the need for more nuanced complexity assessment.