[Paper Review] Graphemic Normalization of the Perso-Arabic Script

Raiomond Doctor, Alexander Gutkin+3 more|arXiv (Cornell University)|Oct 21, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes a graphemic normalization framework for the Perso-Arabic script to address orthographic variation across eight diverse languages, applying normalization techniques to improve NLP performance. It demonstrates statistically significant gains in machine translation and language modeling across all languages, highlighting the necessity of handling regional script variations for low-resource NLP systems.

ABSTRACT

Since its original appearance in 1991, the Perso-Arabic script representation in Unicode has grown from 169 to over 440 atomic isolated characters spread over several code pages representing standard letters, various diacritics and punctuation for the original Arabic and numerous other regional orthographic traditions. This paper documents the challenges that Perso-Arabic presents beyond the best-documented languages, such as Arabic and Persian, building on earlier work by the expert community. We particularly focus on the situation in natural language processing (NLP), which is affected by multiple, often neglected, issues such as the use of visually ambiguous yet canonically nonequivalent letters and the mixing of letters from different orthographies. Among the contributing conflating factors are the lack of input methods, the instability of modern orthographies, insufficient literacy, and loss or lack of orthographic tradition. We evaluate the effects of script normalization on eight languages from diverse language families in the Perso-Arabic script diaspora on machine translation and statistical language modeling tasks. Our results indicate statistically significant improvements in performance in most conditions for all the languages considered when normalization is applied. We argue that better understanding and representation of Perso-Arabic script variation within regional orthographic traditions, where those are present, is crucial for further progress of modern computational NLP techniques especially for languages with a paucity of resources.

Motivation & Objective

• To address the challenge of orthographic inconsistency in the Perso-Arabic script across multiple regional languages beyond Arabic and Persian.
• To investigate how graphemic normalization affects NLP tasks such as machine translation and statistical language modeling.
• To evaluate the impact of normalization on low-resource languages with unstable or under-documented orthographies.
• To advocate for better integration of regional orthographic traditions in computational NLP pipelines.

Proposed method

• The authors collect and standardize graphemic representations from diverse Perso-Arabic script variants across eight languages from different language families.
• They apply normalization rules to resolve visually ambiguous yet canonically distinct characters and to unify diacritics and punctuation across orthographic traditions.
• Normalization is applied to training data for both machine translation and statistical language modeling tasks.
• The method evaluates performance differences between normalized and raw input data using standard NLP metrics.
• The approach accounts for input method limitations, literacy gaps, and orthographic instability in low-resource settings.
• The study uses a comparative framework to measure performance improvements across multiple languages and tasks.

Experimental results

Research questions

• RQ1 How does graphemic normalization affect machine translation performance across diverse Perso-Arabic script languages?
• RQ2 To what extent does normalization improve statistical language modeling in low-resource NLP settings?
• RQ3 What is the impact of orthographic variation—especially from mixed or unstable orthographies—on NLP model performance?
• RQ4 How do regional script differences, including diacritics and letter forms, affect NLP outcomes when not normalized?

Key findings

• Graphemic normalization led to statistically significant improvements in machine translation performance across all eight languages studied.
• Statistical language modeling accuracy improved significantly after normalization, particularly in low-resource settings with unstable orthographies.
• The gains were most pronounced in languages with high orthographic variation and limited digital resources.
• Normalization reduced the impact of visually similar but canonically distinct characters, improving model generalization.
• The study confirms that script-level normalization is essential for consistent and reliable NLP performance across the Perso-Arabic script diaspora.
• The results underscore the importance of incorporating regional orthographic traditions into NLP systems to enhance robustness and accuracy.