[Paper Review] Discriminating between Indo-Aryan Languages Using SVM Ensembles

Alina Maria Ciobanu, Marcos Zampieri+3 more|arXiv (Cornell University)|Jul 9, 2018

Authorship Attribution and Profiling31 references4 citations

TL;DR

This paper presents an SVM ensemble system for discriminating between five closely related Indo-Aryan languages—Hindi, Braj Bhasha, Awadhi, Bhojpuri, and Magahi—using character n-grams (bigrams, trigrams, 4-grams) and word-level features. The system achieved an 88.95% F1 score, ranking 3rd in the VarDial 2018 ILI shared task, with error analysis revealing challenges in short and named-entity-laden texts and high confusion between Bhojpuri and Hindi due to linguistic similarity.

ABSTRACT

In this paper we present a system based on SVM ensembles trained on characters and words to discriminate between five similar languages of the Indo-Aryan family: Hindi, Braj Bhasha, Awadhi, Bhojpuri, and Magahi. We investigate the performance of individual features and combine the output of single classifiers to maximize performance. The system competed in the Indo-Aryan Language Identification (ILI) shared task organized within the VarDial Evaluation Campaign 2018. Our best entry in the competition, named ILIdentification, scored 88:95% F1 score and it was ranked 3rd out of 8 teams.

Motivation & Objective

• To address the challenge of distinguishing between highly similar Indo-Aryan languages, such as Hindi, Braj Bhasha, Awadhi, Bhojpuri, and Magahi, which are difficult for standard language identification systems.
• To improve classification performance through an ensemble of SVM classifiers trained on diverse character and word-level features.
• To evaluate the system in the VarDial 2018 Indo-Aryan Language Identification (ILI) shared task, using a dataset of 90,000 documents from OCR-processed and proofread texts.
• To conduct a detailed error analysis to identify patterns in misclassification, particularly for short sentences and named entities.
• To explore future improvements, including multi-label classification and comparison with state-of-the-art back-off models.

Proposed method

• An ensemble of SVM classifiers was constructed using the Scikit-learn library, with each base classifier trained on a distinct feature set: character bigrams, trigrams, and 4-grams.
• LinearSVC with a linear kernel was used for each individual classifier, chosen for scalability with large datasets.
• The ensemble combined predictions via a majority rule voting strategy (VotingClassifier) to improve robustness and overall performance.
• Feature engineering focused on character-level n-grams to capture phonological and orthographic differences among the closely related languages.
• The system was trained exclusively on the official ILI shared task dataset, without external data or resources.
• Performance was evaluated using weighted F1 score on the test set, with confusion matrices and error analysis conducted on the development set.

Experimental results

Research questions

• RQ1 How effective is an SVM ensemble with character n-gram features in distinguishing between five closely related Indo-Aryan languages?
• RQ2 Which combination of character-level features (bigrams, trigrams, 4-grams) yields optimal performance for this language identification task?
• RQ3 What are the primary sources of misclassification in this multilingual setting, particularly for short or named-entity-laden texts?
• RQ4 To what extent do linguistic similarities between Bhojpuri and Hindi hinder classifier performance, and can this be mitigated?
• RQ5 Can a multi-label classification approach better handle ambiguous or overlapping language instances?

Key findings

• The SVM ensemble system achieved a weighted F1 score of 88.95% on the test set, ranking 3rd out of 8 teams in the ILI shared task.
• The optimal feature combination was found to be character bigrams, trigrams, and 4-grams, based on development set performance.
• Braj Bhasha was the most accurately classified language, while Awadhi had the lowest number of correctly classified instances.
• A total of 94 Bhojpuri instances were misclassified as Hindi, and 176 Hindi instances were misclassified as Bhojpuri, indicating high mutual confusion.
• Approximately 10% of misclassified instances were extremely short (1–3 words), and many others contained named entities, which confused the classifier.
• Native speakers noted that the absence of light verbs in Bhojpuri, compared to their frequent use in Hindi, is a key linguistic distinction, though it was not fully captured by the model.