[Paper Review] Short Text Language Identification for Under Resourced Languages

Bernardt Duvenhage|arXiv (Cornell University)|Nov 18, 2019

Authorship Attribution and Profiling22 references4 citations

TL;DR

This paper proposes a hierarchical naive Bayes and lexicon-based classifier for short text language identification in under-resourced South African languages, achieving 98.70% accuracy on the DSL 2017 dataset. The model leverages language family grouping and lexicon support to improve performance on similar languages, outperforming existing methods on short, informal texts with high efficiency.

ABSTRACT

The paper presents a hierarchical naive Bayesian and lexicon based classifier for short text language identification (LID) useful for under resourced languages. The algorithm is evaluated on short pieces of text for the 11 official South African languages some of which are similar languages. The algorithm is compared to recent approaches using test sets from previous works on South African languages as well as the Discriminating between Similar Languages (DSL) shared tasks' datasets. Remaining research opportunities and pressing concerns in evaluating and comparing LID approaches are also discussed.

Motivation & Objective

• Address the challenge of accurate language identification in short, informal texts for under-resourced South African languages with high linguistic similarity.
• Improve performance on similar languages (e.g., Nguni and Sotho groups) through hierarchical classification and lexicon integration.
• Develop a lightweight, efficient model suitable for deployment in low-resource NLP pipelines with limited training data.
• Evaluate the model against recent state-of-the-art approaches on standardized benchmarks, including DSL 2015 and DSL 2017 datasets.
• Explore the role of lexicon support and data augmentation to reduce dependency on large annotated corpora.

Proposed method

• Proposes a two-stage hierarchical classifier: first grouping languages by family (e.g., Nguni, Sotho), then identifying the exact language using a naive Bayes model.
• Integrates a lexicon-based component to enhance discrimination between similar languages, particularly in low-resource settings.
• Employs character-level n-grams and language family features as input to the naive Bayes classifier.
• Uses a stacked model architecture combining a naive Bayes classifier with a lexicon module, with ablation studies on lexicon dropout (50%) to assess impact.
• Implements the model in Python using scikit-learn with default hyperparameters and a two-stage learning rate schedule (0.001 then 0.0001).
• Evaluates performance on three benchmark datasets: NCHLT, DSL 2015, and DSL 2017, using standard accuracy metrics.

Experimental results

Research questions

• RQ1 How effective is a hierarchical naive Bayes and lexicon-based approach for short text language identification in under-resourced, similar South African languages?
• RQ2 To what extent does lexicon support improve classification accuracy, especially for closely related languages?
• RQ3 How does the model’s performance compare to state-of-the-art methods like fasttext, SVMs, and RNN ensembles on standardized short-text benchmarks?
• RQ4 What is the inference speed and computational efficiency of the proposed model compared to deep learning baselines?
• RQ5 Can data augmentation and self-supervised lexicon maintenance reduce dependency on large annotated datasets in low-resource settings?

Key findings

• The stacked model combining naive Bayes and lexicon (NB+Lex) achieved 98.70% accuracy on the DSL 2017 dataset, outperforming all reported baselines on this benchmark.
• The lexicon-only model achieved 93.56% accuracy on DSL 2017, demonstrating significant value in distinguishing similar languages when properly supported.
• The model achieved 96.12% accuracy on the NCHLT dataset, surpassing the standalone naive Bayes (94.36%) and NB+NB (94.41%) variants.
• The proposed model processed 7.4k inference requests per second on the NCHLT dataset, outperforming slower models like the BRNN (0.75/s) and matching the fasttext (44k/s) in speed.
• Ablation with 50% lexicon dropout reduced accuracy to 96.21% on DSL 2017, confirming that lexicon support is a key contributor to performance.
• The model’s performance was highly dependent on lexicon quality, with a 50% dropout reducing accuracy by 2.5 percentage points, indicating strong reliance on lexicon features.