[Paper Review] Handshakes AI Research at CASE 2021 Task 1: Exploring different approaches for multilingual tasks

Vivek Kalyan, Paul Tan+2 more|arXiv (Cornell University)|Oct 29, 2021

Computational and Text Analysis Methods4 citations

TL;DR

This paper presents a multilingual approach to socio-political and crisis event detection across document, sentence, coreference, and event extraction tasks using pre-trained multilingual transformers. By training a single XLM-RoBERTa model on a combined multilingual dataset and leveraging cross-lingual transfer with word-level translation and LaBSE embeddings, the authors achieved state-of-the-art performance, particularly improving zero-shot generalization to unseen languages like Hindi.

ABSTRACT

The aim of the CASE 2021 Shared Task 1 (H\\"urriyeto\\u{g}lu et al., 2021) was to detect and classify socio-political and crisis event information at document, sentence, cross-sentence, and token levels in a multilingual setting, with each of these subtasks being evaluated separately in each test language. Our submission contained entries in all of the subtasks, and the scores obtained validated our research finding: That the multilingual aspect of the tasks should be embraced, so that modeling and training regimes use the multilingual nature of the tasks to their mutual benefit, rather than trying to tackle the different languages separately. Our code is available at https://github.com/HandshakesByDC/case2021/

Motivation & Objective

• To develop a unified multilingual system for detecting and classifying socio-political and crisis events across multiple NLP tasks.
• To investigate whether joint training on multilingual data improves performance over language-specific fine-tuning.
• To enable zero-shot classification on low-resource languages like Hindi without task-specific training data.
• To evaluate the effectiveness of cross-lingual transfer via word-to-word translation and multilingual sentence embeddings.
• To determine whether shared modeling across languages enhances performance on both seen and unseen languages.

Proposed method

• Trained a single XLM-RoBERTa-base model on a combined multilingual dataset spanning English, Spanish, Portuguese, and Hindi.
• Used LaBSE and LASER embeddings to enable cross-lingual transfer for low-resource languages.
• Applied word-by-word translation using bilingual dictionaries to generate synthetic training data for secondary languages.
• Employed Viterbi decoding with constrained transition probabilities to maintain BIO tagging consistency in event extraction.
• Combined training data from all languages into a single dataset to improve model generalization.
• Fine-tuned the model with a classification head on top of pooled transformer outputs for binary classification tasks.

Experimental results

Research questions

• RQ1 Can a single multilingual model trained on combined data outperform language-specific models in multilingual event detection tasks?
• RQ2 To what extent does cross-lingual transfer via translated data improve performance on low-resource languages like Hindi?
• RQ3 How does joint multilingual training affect performance on high-resource languages like English compared to monolingual fine-tuning?
• RQ4 Does the use of multilingual sentence embeddings (e.g., LaBSE, M-USE) improve zero-shot generalization to unseen languages?
• RQ5 What is the impact of constrained decoding (e.g., Viterbi with BIO constraints) on event extraction quality in multilingual settings?

Key findings

• The best-performing model for Subtask 2 (sentence classification) achieved a competition F1 score of 0.8506 on Portuguese, outperforming language-specific baselines.
• Training on a combined multilingual dataset improved performance on Spanish and Portuguese, while English showed minimal gains due to abundant existing data.
• For Subtask 4 (event extraction), the model using both Viterbi decoding and word-to-word translated data achieved a validation F1 of 82.53 on English, 62.17 on Spanish, and 72.75 on Portuguese.
• The model achieved a final competition ranking of 2nd in both Spanish and Portuguese for Subtask 4, indicating strong generalization to secondary languages.
• Performance on the test set was lower than on the validation set by approximately 5%, suggesting potential distributional shifts or data leakage in validation splits.
• Adding translated secondary language data increased convergence time by ~2x but improved F1 scores, indicating effective data augmentation without performance degradation.