[Paper Review] LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation

Gustavo Aguilar, Sudipta Kar+1 more|arXiv (Cornell University)|May 9, 2020

Natural Language Processing Techniques39 references4 citations

TL;DR

This paper introduces LinCE, a centralized benchmark for evaluating NLP models on linguistic code-switching across four language pairs (Spanish-English, Hindi-English, Nepali-English, and Modern Standard Arabic-Egyptian Arabic) and four tasks (language identification, POS tagging, NER, and sentiment analysis). It provides standardized, stratified data splits and integrates strong baselines using BiLSTM, ELMo, and multilingual BERT, with multilingual BERT outperforming others by an average margin of ~6% across tasks.

ABSTRACT

Recent trends in NLP research have raised an interest in linguistic code-switching (CS); modern approaches have been proposed to solve a wide range of NLP tasks on multiple language pairs. Unfortunately, these proposed methods are hardly generalizable to different code-switched languages. In addition, it is unclear whether a model architecture is applicable for a different task while still being compatible with the code-switching setting. This is mainly because of the lack of a centralized benchmark and the sparse corpora that researchers employ based on their specific needs and interests. To facilitate research in this direction, we propose a centralized benchmark for Linguistic Code-switching Evaluation (LinCE) that combines ten corpora covering four different code-switched language pairs (i.e., Spanish-English, Nepali-English, Hindi-English, and Modern Standard Arabic-Egyptian Arabic) and four tasks (i.e., language identification, named entity recognition, part-of-speech tagging, and sentiment analysis). As part of the benchmark centralization effort, we provide an online platform at ritual.uh.edu/lince, where researchers can submit their results while comparing with others in real-time. In addition, we provide the scores of different popular models, including LSTM, ELMo, and multilingual BERT so that the NLP community can compare against state-of-the-art systems. LinCE is a continuous effort, and we will expand it with more low-resource languages and tasks.

Motivation & Objective

• Address the lack of a unified benchmark for evaluating code-switching NLP models across diverse language pairs and tasks.
• Overcome inconsistencies in existing datasets, such as data leakage and imbalanced splits, which hinder fair model comparison.
• Facilitate direct, real-time comparison of model performance across multiple tasks and language pairs through an online platform.
• Provide strong, standardized baselines using popular models like BiLSTM, ELMo, and multilingual BERT to establish performance baselines.
• Enable future expansion of the benchmark to include low-resource languages and new NLP tasks as they emerge.

Proposed method

• Consolidated ten existing public code-switched datasets into a unified benchmark covering four language pairs and four NLP tasks.
• Implemented a novel stratification process using up to three criteria—LID labels, task-specific labels, and sentence length—to ensure balanced and representative train/valid/test splits.
• Identified and corrected five major data issues in original datasets, including label leakage and distributional skew, in nine out of ten datasets.
• Provided standardized, official splits that preserve the full corpus distribution, improving reproducibility and fairness in evaluation.
• Deployed an online platform at ritual.uh.edu/lince for real-time model submission and comparison across the community.
• Trained and evaluated three strong baseline models: BiLSTM with word and character embeddings, ELMo with contextual representations, and multilingual BERT fine-tuned on each task.

Experimental results

Research questions

• RQ1 To what extent do existing code-switching datasets suffer from data leakage and distributional imbalances that compromise model evaluation?
• RQ2 Can a centralized benchmark with standardized, stratified splits improve the reliability and comparability of code-switching NLP model evaluations?
• RQ3 How do state-of-the-art models (BiLSTM, ELMo, multilingual BERT) perform across diverse code-switched language pairs and tasks?
• RQ4 What is the performance gap between pre-trained models like ELMo and multilingual BERT in code-switched settings, and what factors contribute to it?
• RQ5 How do task difficulty and language pair typology (e.g., high vs. low resource, script similarity) affect model performance in code-switched NLP tasks?

Key findings

• Multilingual BERT outperformed both ELMo and BiLSTM across all tasks, with an average performance gap of approximately 6% over ELMo.
• The performance gap between ELMo and multilingual BERT was largest in NER (8–10%) and sentiment analysis (18%), indicating greater challenges in semantic understanding tasks.
• Language identification was the most robust task, with the highest accuracy across language pairs, while sentiment analysis and NER proved significantly more challenging.
• The MSA-Egyptian Arabic pair posed the greatest challenge for language identification, likely due to high lexical overlap between the languages.
• The new stratified splits successfully preserved the distribution of the original corpora, unlike the original splits which often suffered from data leakage or imbalance.
• The benchmark revealed that model generalization across language pairs and tasks remains limited, even with strong pre-trained models, highlighting the need for further research in multilingual code-switched NLP.