[Paper Review] The ParlaSent Multilingual Training Dataset for Sentiment Identification in Parliamentary Proceedings

Michal Mocht̆ak, Peter Rupnik+1 more|arXiv (Cornell University)|Sep 18, 2023

Natural Language Processing TechniquesComputer Science3 citations

TL;DR

This paper introduces ParlaSent, a multilingual, manually annotated dataset of 13,000 parliamentary sentences across five European languages (Czech, English, Serbo-Croatian, Slovak, Slovene) from seven countries, annotated for six-level sentiment. It demonstrates that fine-tuning a multilingual XLM-RoBERTa model on this data—especially with additional pre-training on 1.72 billion domain-specific parliamentary words—significantly improves sentiment classification performance, even generalizing well to unseen languages and parliaments.

ABSTRACT

The paper presents a new training dataset of sentences in 7 languages, manually annotated for sentiment, which are used in a series of experiments focused on training a robust sentiment identifier for parliamentary proceedings. The paper additionally introduces the first domain-specific multilingual transformer language model for political science applications, which was additionally pre-trained on 1.72 billion words from parliamentary proceedings of 27 European parliaments. We present experiments demonstrating how the additional pre-training on parliamentary data can significantly improve the model downstream performance, in our case, sentiment identification in parliamentary proceedings. We further show that our multilingual model performs very well on languages not seen during fine-tuning, and that additional fine-tuning data from other languages significantly improves the target parliament's results. The paper makes an important contribution to multiple disciplines inside the social sciences, and bridges them with computer science and computational linguistics. Lastly, the resulting fine-tuned language model sets up a more robust approach to sentiment analysis of political texts across languages, which allows scholars to study political sentiment from a comparative perspective using standardized tools and techniques.

Motivation & Objective

• Address the scarcity of high-quality, domain-specific sentiment datasets for low-resourced European languages in political science.
• Bridge the gap between computational linguistics and social sciences by enabling robust, standardized sentiment analysis in comparative parliamentary research.
• Provide a practical, resource-aware framework for social scientists to build and train sentiment classifiers on political discourse.
• Demonstrate the benefits of multilingual and domain-adapted pre-training for improving sentiment classification in low-resource political text settings.

Proposed method

• Collect and manually annotate 2,600 sentences per parliamentary group (5 total groups) from 27 European parliaments, using a six-level sentiment schema.
• Pre-train a multilingual XLM-RoBERTa model on 1.72 billion domain-specific words from parliamentary proceedings across multiple languages.
• Fine-tune the pre-trained model on the ParlaSent dataset using a cross-entropy loss for sequence classification.
• Train both monolingual and multilingual models, comparing performance when training on target-language data only versus on combined multilingual data.
• Evaluate model performance on unseen languages and parliaments using held-out test sets from the UK and BCS (Bosnia-Herzegovina, Croatia, Serbia) parliamentary groups.
• Use inter-annotator agreement (Cohen’s Kappa ≈ 0.65) to validate annotation consistency across languages and parliaments.

Experimental results

Research questions

• RQ1 Does additional pre-training on raw parliamentary text improve the performance of a transformer-based sentiment classifier?
• RQ2 How well do multilingual models generalize to unseen languages and parliaments not included in the training data?
• RQ3 Is training a sentiment classifier on multilingual data from multiple parliaments more effective than training on monolingual data from a single parliament?
• RQ4 To what extent does inter-annotator agreement remain high across different languages and parliaments, indicating reliable annotation guidelines?

Key findings

• Additional pre-training on 1.72 billion domain-specific words from parliamentary proceedings significantly improves downstream sentiment classification performance.
• The multilingual XLM-R-Parla model achieves strong performance on unseen languages and parliaments, with minimal performance drop, indicating robust cross-lingual generalization.
• Multilingual models trained on four times more data from other languages outperform monolingual models trained only on target-language data in all parliaments except Czech, where performance is comparable.
• The XLM-R-ParlaSent model, fine-tuned on the ParlaSent dataset, achieves a macro-F1 score of 0.665 on the UK test set and 0.708 on the BCS test set, demonstrating strong zero-shot generalization.
• Inter-annotator agreement (Cohen’s Kappa ≈ 0.65) indicates that the annotation guidelines are sufficiently detailed and consistent across languages and parliaments.
• The model performs best when trained on multilingual data, suggesting that cross-parliamentary and cross-linguistic patterns in sentiment expression are beneficial for classification.