[Paper Review] German Dialect Identification Using Classifier Ensembles

Alina Maria Ciobanu, Shervin Malmasi+1 more|arXiv (Cornell University)|Jul 22, 2018

Authorship Attribution and Profiling12 references4 citations

TL;DR

This paper presents an SVM-based ensemble classifier for German dialect identification, leveraging character and word n-grams with TF-IDF weighting to distinguish four Swiss-German dialects (Basel, Bern, Lucerne, Zurich). The system achieved a 62.03% F1 score on the test set, ranking third out of eight teams in the VarDial 2018 shared task, with character 4-grams and an ensemble of four classifiers yielding optimal performance.

ABSTRACT

In this paper we present the GDI_classification entry to the second German Dialect Identification (GDI) shared task organized within the scope of the VarDial Evaluation Campaign 2018. We present a system based on SVM classifier ensembles trained on characters and words. The system was trained on a collection of speech transcripts of five Swiss-German dialects provided by the organizers. The transcripts included in the dataset contained speakers from Basel, Bern, Lucerne, and Zurich. Our entry in the challenge reached 62.03% F1-score and was ranked third out of eight teams.

Motivation & Objective

• To develop a robust system for classifying Swiss-German dialects from speech transcripts.
• To improve dialect identification performance through ensemble learning with diverse feature types.
• To evaluate the effectiveness of character and word n-grams with TF-IDF weighting in dialect classification.
• To rank among top systems in the GDI shared task 2018, building on prior work in author profiling and dialect identification.

Proposed method

• An ensemble of 14 individual SVM classifiers was trained, each using a distinct feature type: character n-grams (n=1 to 8), word n-grams (n=1 to 3), and word k-skip bigrams (k=1 to 3).
• LinearSVC with a linear kernel and L2 regularization was used for all base classifiers, with hyperparameter C tuned via grid search over {10⁻³, ..., 10³}.
• The ensemble combined predictions using a majority voting strategy, with ties resolved by ascending label order.
• TF-IDF weighting was applied to all feature types to emphasize discriminative n-grams and reduce overrepresentation of frequent terms.
• The optimal ensemble combined four classifiers using character n-grams with n ∈ {2, 3, 4, 5}, selected based on development set performance.
• The system was trained and evaluated on the GDI 2018 dataset, comprising 24,849 transcribed speech instances from Basel, Bern, Lucerne, and Zurich.

Experimental results

Research questions

• RQ1 Can an ensemble of SVM classifiers improve dialect identification accuracy compared to single classifiers?
• RQ2 Which n-gram feature types—character or word-based—yield the highest performance for Swiss-German dialect classification?
• RQ3 Does combining multiple feature types through ensemble learning reduce misclassification, especially between similar dialects?
• RQ4 How does the system perform on unseen dialects, and what is the impact of feature engineering on model generalization?

Key findings

• The best-performing individual classifier used character 4-grams, achieving a 0.621 F1 score on the development set.
• The optimal ensemble of four classifiers using character n-grams (n=2 to 5) achieved a 0.638 F1 score on the development set.
• The final submitted system, using the same four-character n-gram features, achieved a 0.6203 F1 score on the test set, ranking third in the shared task.
• The system significantly outperformed the random baseline, which scored 0.2521 F1 on the test set.
• Confusion matrix analysis revealed that Lucerne (LU) dialects were most frequently misclassified as Bern (BE), indicating high similarity between these dialects.
• The most informative character 4-grams varied by dialect, with distinct patterns such as '|n¨ac| aus |seit|nei|' for Bern and '|uns|aabe|kha|rlig|' for Basel.