[Paper Review] Automatic Difficulty Classification of Arabic Sentences

Nouran Khallaf, Serge Sharoff|arXiv (Cornell University)|Mar 7, 2021

Text Readability and Simplification32 references8 citations

TL;DR

This paper presents the first sentence-level difficulty classifier for Modern Standard Arabic, using fine-tuned Arabic-BERT to predict CEFR proficiency levels (A, B, C) or binary difficulty (simple vs. complex). It achieves an F1-score of 0.80 for 3-way CEFR classification and 0.94 for binary classification, demonstrating that contextual embeddings outperform traditional linguistic features and baseline models.

ABSTRACT

In this paper, we present a Modern Standard Arabic (MSA) Sentence difficulty classifier, which predicts the difficulty of sentences for language learners using either the CEFR proficiency levels or the binary classification as simple or complex. We compare the use of sentence embeddings of different kinds (fastText, mBERT , XLM-R and Arabic-BERT), as well as traditional language features such as POS tags, dependency trees, readability scores and frequency lists for language learners. Our best results have been achieved using fined-tuned Arabic-BERT. The accuracy of our 3-way CEFR classification is F-1 of 0.80 and 0.75 for Arabic-Bert and XLM-R classification respectively and 0.71 Spearman correlation for regression. Our binary difficulty classifier reaches F-1 0.94 and F-1 0.98 for sentence-pair semantic similarity classifier.

Motivation & Objective

• To develop the first sentence-level difficulty classifier for Modern Standard Arabic, addressing a gap in NLP for Arabic language learning.
• To evaluate the effectiveness of various sentence representation methods—ranging from traditional linguistic features to state-of-the-art neural embeddings—on Arabic sentence difficulty prediction.
• To enable automatic text simplification and curriculum-based assessment by accurately identifying complex sentences unsuitable for beginner learners.
• To create a novel, high-quality dataset of sentence-level difficulty annotations for Arabic, derived from multiple sources including learner corpora and textbooks.
• To explore transferability of models across different corpora and assess the interpretability of deep learning models through linguistic feature probing.

Proposed method

• Constructed a new dataset of 22,740 sentences annotated with CEFR levels (A, B, C) by mapping document-level annotations from learner corpora and textbooks to individual sentences using heuristics and re-annotation.
• Extracted a comprehensive set of linguistic features, including POS tags, dependency tree structures, word frequency (from Buckwalter & Parkinson and Al-Kitaab word lists), and readability scores (e.g., Dawood, Al-Heeti, AARI-Base).
• Evaluated multiple sentence embedding models: fastText, mBERT, XLM-R, and Arabic-BERT, fine-tuning the latter on the difficulty classification task.
• Formulated the problem as both multi-class classification (3-way CEFR: A, B, C) and binary classification (simple vs. complex), and also as regression to predict continuous difficulty scores.
• Trained and compared models using traditional machine learning (SVM, logistic regression) and deep learning approaches, with ablation studies to assess feature contributions.
• Applied a multi-classifier agreement strategy (Di Bari et al.) for dataset cleaning, using SVM, random forest, and logistic regression to detect and correct labeling errors via expert validation.

Experimental results

Research questions

• RQ1 Can fine-tuned Arabic-BERT outperform other pre-trained language models and traditional linguistic features in classifying the difficulty of Arabic sentences?
• RQ2 How effective are traditional readability formulas and frequency-based features in predicting sentence difficulty for Arabic language learners?
• RQ3 To what extent can models trained on one corpus generalize to a different evaluation corpus, and how does this vary between feature-based and neural network-based models?
• RQ4 Does the integration of linguistic features improve the interpretability of deep learning models in sentence difficulty classification?
• RQ5 How well does the model distinguish between lower proficiency levels (A/B) versus higher levels (C), and what are the limitations in this separation?

Key findings

• Fine-tuned Arabic-BERT achieved the highest performance with an F1-score of 0.80 on 3-way CEFR classification, outperforming XLM-R (F1 0.75) and other models.
• The binary difficulty classifier reached an F1-score of 0.94 using sentence-pair semantic similarity, indicating strong performance in distinguishing simple from complex sentences.
• The regression task achieved a Spearman correlation of 0.71, showing that the model can predict continuous difficulty levels with moderate accuracy.
• Traditional linguistic features such as POS, frequency lists, and syntactic structures contributed to model interpretability but were less effective than deep learning embeddings alone.
• Transfer learning performance was higher for feature-based models than for BERT-based models, suggesting that BERT models learn more task-specific representations.
• The model shows promise for educational applications, particularly in filtering out C-level sentences for undergraduate Arabic curricula, supporting text selection and curriculum gap analysis.