[Paper Review] Classifier Ensembles for Dialect and Language Variety Identification

Liviu P. Dinu, Alina Maria Ciobanu+2 more|arXiv (Cornell University)|Aug 14, 2018

Authorship Attribution and Profiling16 references3 citations

TL;DR

This paper presents ensemble-based SVM systems for dialect and language variety identification, using TF-IDF weighted character n-grams, word n-grams, and word k-skip bigrams. The method achieved 0.596 F1 on Dutch/Flemish discrimination and 0.500 F1 on Arabic dialect identification, outperforming baseline but falling short of top-tier systems in both shared tasks at VarDial 2018.

ABSTRACT

In this paper we present ensemble-based systems for dialect and language variety identification using the datasets made available by the organizers of the VarDial Evaluation Campaign 2018. We present a system developed to discriminate between Flemish and Dutch in subtitles and a system trained to discriminate between four Arabic dialects: Egyptian, Levantine, Gulf, North African, and Modern Standard Arabic in speech broadcasts. Finally, we compare the performance of these two systems with the other systems submitted to the Discriminating between Dutch and Flemish in Subtitles (DFS) and the Arabic Dialect Identification (ADI) shared tasks at VarDial 2018.

Motivation & Objective

• To develop and evaluate ensemble-based machine learning systems for discriminating between Dutch and Flemish in subtitles.
• To build and test a multilingual ensemble system for identifying four Arabic dialects and Modern Standard Arabic in speech broadcasts.
• To compare the performance of the proposed ensemble approach against other systems in the VarDial 2018 shared tasks.
• To analyze the limitations of the current system and identify areas for improvement through error and feature analysis.
• To contribute to the understanding of language variety identification in low-resource and closely related language pairs.

Proposed method

• Employed a voting ensemble of multiple linear SVM classifiers, each trained on a different feature type: character n-grams (n=1 to 8), word n-grams (n=1 to 3), and word k-skip bigrams (k=1 to 3).
• Used TF-IDF vectorization to represent text features, ensuring feature importance was weighted by term frequency and inverse document frequency.
• Combined individual classifier predictions using majority rule voting with uniform weights, resolving ties via lexicographic label ordering.
• Trained models on official training data from the VarDial 2018 shared tasks: DFS (Dutch/Flemish) and ADI (Arabic dialects and MSA).
• Evaluated performance on official development and test sets, using macro F1 as the primary metric.
• Conducted confusion matrix analysis to diagnose misclassification patterns and inform future improvements.

Experimental results

Research questions

• RQ1 How effective is an ensemble of SVM classifiers with diverse n-gram features in distinguishing between Dutch and Flemish in subtitle data?
• RQ2 To what extent can the same ensemble framework identify four Arabic dialects and Modern Standard Arabic in speech transcripts?
• RQ3 How does the performance of the proposed ensemble compare to state-of-the-art systems in the DFS and ADI shared tasks at VarDial 2018?
• RQ4 What are the primary sources of error in the system’s predictions, particularly in distinguishing between similar dialects like Levantine and North African Arabic?
• RQ5 Can feature importance and error analysis inform improvements for future language variety identification systems?

Key findings

• The ensemble system achieved a macro F1 score of 0.596 on the DFS shared task, ranking sixth among 12 teams and significantly outperforming the 0.500 random baseline.
• For the ADI shared task, the system achieved a macro F1 score of 0.500, matching the random baseline and indicating limited discriminative power for Arabic dialects.
• The system performed best at identifying Egyptian and North African Arabic dialects, with the highest confusion occurring between Levantine and North African dialects.
• On the DFS task, the system showed slightly better performance in identifying Flemish (BEL) than Dutch (DUT), suggesting some sensitivity to regional linguistic markers.
• Despite strong performance in similar tasks (e.g., ILI and GDI shared tasks), the system underperformed in the DFS and ADI tasks, indicating domain-specific challenges.
• Error analysis and feature importance analysis are ongoing to improve model generalization and robustness in future iterations.