[Paper Review] kk2018 at SemEval-2020 Task 9: Adversarial Training for Code-Mixing Sentiment Classification

Jiaxiang Liu, Xuyi Chen+6 more|arXiv (Cornell University)|Sep 8, 2020

Natural Language Processing Techniques22 references4 citations

TL;DR

This paper proposes adversarial training with a multilingual XLM-R backbone for code-mixing sentiment classification, achieving state-of-the-art performance on the SemEval-2020 Task 9 Hindi-English dataset. It demonstrates that domain transfer from a strong monolingual model (ERNIE) provides a surprisingly strong baseline, and adversarial training significantly improves generalization and test performance, culminating in a 1st-place F1 score of 0.7526 via ensemble methods.

ABSTRACT

Code switching is a linguistic phenomenon that may occur within a multilingual setting where speakers share more than one language. With the increasing communication between groups with different languages, this phenomenon is more and more popular. However, there are little research and data in this area, especially in code-mixing sentiment classification. In this work, the domain transfer learning from state-of-the-art uni-language model ERNIE is tested on the code-mixing dataset, and surprisingly, a strong baseline is achieved. Furthermore, the adversarial training with a multi-lingual model is used to achieve 1st place of SemEval-2020 Task 9 Hindi-English sentiment classification competition.

Motivation & Objective

• To investigate the effectiveness of domain transfer learning from a monolingual pre-trained model (ERNIE) on a low-resource code-mixed sentiment classification task.
• To evaluate adversarial training as a regularizer to improve model generalization on small, code-mixed datasets.
• To achieve state-of-the-art performance on the SemEval-2020 Task 9 Hindi-English code-mixed sentiment classification benchmark.
• To analyze the impact of hyperparameters in adversarial training on model robustness and performance.

Proposed method

• Fine-tuned the monolingual ERNIE 2.0 model directly on the Hindi-English code-mixed dataset as a strong baseline using domain transfer learning.
• Replaced the backbone with the multilingual XLM-R model and applied adversarial training by perturbing word embeddings within a small ε-ball in the embedding space.
• Introduced a semi-supervised adversarial loss with α=1 to regularize the model during training, improving generalization.
• Used a linear warm-up learning rate schedule with Adam optimizer and trained for 5 epochs, selecting the best model on the development set.
• Constructed an ensemble of 10-fold cross-validation models and diverse hyperparameter settings, combining predictions via probability averaging.
• Applied BPE tokenization to reduce out-of-vocabulary words and improve cross-lingual transfer, especially for Indo-European language pairs.

Experimental results

Research questions

• RQ1 Can a monolingual pre-trained model like ERNIE achieve strong performance on a low-resource code-mixed sentiment classification task through domain transfer?
• RQ2 How does adversarial training with perturbations in the embedding space affect model generalization and performance on code-mixed text?
• RQ3 What is the optimal value of ε in adversarial training for code-mixed sentiment classification, and how does it affect development and test set scores?
• RQ4 Does adversarial training reduce overfitting on small code-mixed datasets, and is this reflected in improved generalization across folds?
• RQ5 Can ensemble methods combining diverse models and hyperparameters further boost performance beyond individual models?

Key findings

• The monolingual ERNIE 2.0 model achieved a strong baseline test F1 score of 0.6725, demonstrating effective domain transfer despite being trained on English-only data.
• Replacing ERNIE with XLM-R improved the test F1 score to 0.7177, indicating that multilingual pre-training is beneficial for code-mixed tasks.
• Adversarial training with ε=5 yielded the highest test F1 score of 0.7349, showing that adversarial regularization enhances model robustness and performance.
• The model with ε=2 achieved the best development F1 score of 0.6723, indicating that moderate perturbation levels optimize generalization.
• The final ensemble model achieved a state-of-the-art F1 score of 0.7526, securing 1st place in the SemEval-2020 Task 9 competition.
• Despite improvements, the neutral class had low precision, suggesting the model remains biased toward detecting clear sentiment cues and struggles with ambiguous or neutral examples.