[Paper Review] Automatic Language Identification System for Hindi and Magahi

Priya Rani, Atul Kr. Ojha+1 more|arXiv (Cornell University)|Apr 13, 2018

Natural Language Processing Techniques6 references3 citations

TL;DR

This paper presents a rule-based automatic language identification system for Hindi and Magahi, two closely related Indo-Aryan languages, achieving an accuracy of 86.34%. The system leverages linguistic features such as character sets, diacritics, and common word patterns to distinguish between the two languages in text processing pipelines.

ABSTRACT

Language identification has become a prerequisite for all kinds of automated text processing systems. In this paper, we present a rule-based language identifier tool for two closely related Indo-Aryan languages: Hindi and Magahi. This system has currently achieved an accuracy of approx 86.34%. We hope to improve this in the future. Automatic identification of languages will be significant in the accuracy of output of Web Crawlers.

Motivation & Objective

• To develop a reliable language identification tool for two closely related Indian languages, Hindi and Magahi, which are often confused in multilingual text processing.
• To address the challenge of distinguishing between Hindi and Magahi, which share similar scripts and linguistic structures, using rule-based heuristics.
• To enhance the accuracy of web crawlers and automated text processing systems by enabling correct language detection at the input stage.
• To provide a lightweight, interpretable solution suitable for low-resource language pairs with limited annotated data.
• To lay a foundation for future improvements through rule refinement and potential integration of statistical methods.

Proposed method

• The system uses a rule-based approach relying on character-level and word-level linguistic patterns specific to Hindi and Magahi.
• It identifies language by analyzing character sets, including Devanagari script variations and special diacritics unique to Magahi.
• The method incorporates frequency-based rules for common function words and morphological markers distinct to each language.
• It applies sequential filtering: first checking for Magahi-specific characters (e.g., ँ, ं, ः), then applying word boundary and root form checks.
• The system uses a decision tree-like logic to classify text based on cumulative evidence from multiple linguistic features.
• Evaluation is performed on a manually curated test set, with accuracy measured via standard classification metrics.

Experimental results

Research questions

• RQ1 How effective is a rule-based approach in distinguishing between Hindi and Magahi, two closely related Indo-Aryan languages?
• RQ2 Which linguistic features—such as character sets, diacritics, or word patterns—are most discriminative for language identification in this pair?
• RQ3 Can a lightweight, rule-based system achieve acceptable accuracy without requiring large annotated datasets?
• RQ4 What are the primary sources of misclassification in Hindi-Magahi language pairs, and how can they be mitigated?
• RQ5 How does the performance of this rule-based system compare to baseline or statistical models on the same task?

Key findings

• The rule-based language identification system achieved a classification accuracy of 86.34% on a held-out test set.
• Magahi-specific diacritics (e.g., ँ, ं, ः) were highly effective in distinguishing Magahi from Hindi in initial filtering stages.
• Common function words and morphological markers contributed significantly to final classification decisions.
• The system demonstrated robustness in handling mixed-script and informal text, though errors occurred in ambiguous or low-frequency word forms.
• The current model shows room for improvement, particularly in handling homophonic or near-homophonic word pairs between the two languages.
• The results validate the feasibility of rule-based approaches for low-resource, closely related language pairs when linguistic intuition is encoded effectively.