[Paper Review] LIDE: Language Identification from Text Documents

Priyank Mathur, Arkajyoti Misra+1 more|arXiv (Cornell University)|Jan 13, 2017

Authorship Attribution and Profiling5 references3 citations

TL;DR

LIDE is a deep learning-based language identification system that leverages bidirectional RNNs and character-level n-grams to achieve 95.12% accuracy on the DSL 2015 benchmark, outperforming industry-standard APIs and traditional models in distinguishing highly similar languages such as South Western Slavic and Ibero-Romance variants.

ABSTRACT

The increase in the use of microblogging came along with the rapid growth on short linguistic data. On the other hand deep learning is considered to be the new frontier to extract meaningful information out of large amount of raw data in an automated manner. In this study, we engaged these two emerging fields to come up with a robust language identifier on demand, namely Language Identification Engine (LIDE). As a result, we achieved 95.12% accuracy in Discriminating between Similar Languages (DSL) Shared Task 2015 dataset, which is comparable to the maximum reported accuracy of 95.54% achieved so far.

Motivation & Objective

• To develop a robust, deep learning-based language identification system capable of distinguishing between highly similar languages that traditional methods struggle with.
• To overcome limitations of tokenization-dependent models by using character-level n-grams, enabling support for languages without word boundaries like Japanese and Chinese.
• To benchmark the proposed model against state-of-the-art commercial and open-source language detection tools on a multilingual dataset with challenging language pairs.
• To improve performance on low-resource and dialectally distinct language pairs through end-to-end learning of linguistic patterns via RNNs.

Proposed method

• Employed character-level n-grams (1–5-grams) as input features to avoid dependency on word segmentation, crucial for languages like Japanese and Chinese.
• Designed a deep neural network architecture combining bidirectional Long Short-Term Memory (LSTM) networks to model sequential dependencies and capture language-specific structural patterns.
• Trained the model end-to-end on a dataset of 18,000 training instances across 13 languages grouped into 7 linguistic families, including challenging pairs like Serbian, Croatian, and Bosnian.
• Used t-SNE visualization to analyze feature space and confirm that similar languages cluster together, validating the model’s ability to learn subtle linguistic distinctions.
• Applied ensemble learning by combining multiple RNN models to improve generalization and robustness across diverse language families.
• Performed hyperparameter tuning using a development set (devel.txt) and evaluated final performance on the official test set (test-gold.txt).

Experimental results

Research questions

• RQ1 Can a deep learning model achieve superior performance in language identification compared to traditional n-gram and probabilistic models on similar language pairs?
• RQ2 To what extent can character-level n-grams and RNNs capture subtle linguistic differences between closely related languages such as Ibero-Romance and South Western Slavic variants?
• RQ3 How does the proposed model perform relative to commercial and open-source language detection APIs on a standardized benchmark with mixed-language and dialectal instances?
• RQ4 Can end-to-end learning with RNNs overcome the limitations of rule-based or frequency-based models in low-resource and morphologically complex language settings?
• RQ5 What role does architectural design—particularly bidirectional RNNs—play in distinguishing languages with overlapping orthographic and grammatical features?

Key findings

• LIDE achieved 95.12% accuracy on the DSL 2015 test set, ranking first among all evaluated systems and closely approaching the state-of-the-art result of 95.54%.
• The model significantly outperformed Google Translate API (89%) and Yandex Translator API (79%) in distinguishing South Western Slavic languages, where the latter two achieved only 38% accuracy on this challenging subgroup.
• LIDE demonstrated strong generalization on the Astronesian group (Indonesian and Malay), achieving higher accuracy than Yandex (62.75%) and Rosette (86%) on this complex subgroup.
• The confusion matrices revealed that commercial APIs like Google Translate and Yandex failed to differentiate between dialects such as Brazilian and European Portuguese, while LIDE correctly classified them.
• Langid.py, a widely used open-source tool, achieved 80% accuracy, but still underperformed LIDE, highlighting the advantage of deep learning over traditional Naive Bayes approaches.
• The t-SNE visualization confirmed that while similar languages cluster together, the model successfully learned to separate them, especially in the Ibero-Romance and South Western Slavic groups.