[论文解读] XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond

Francesco Barbieri, Luis Espinosa-Anke+1 more|arXiv (Cornell University)|Apr 25, 2021

Sentiment Analysis and Opinion Mining被引用 134

一句话总结

XLM-T 是一个专门面向 Twitter 的多语言模型，建立在 XLM-R 基础之上，在 198M 条推文、30+ 种语言上进行训练，并在统一的多语言情感基准和跨语言迁移任务上进行评估，提供分析和微调的起步工具。

ABSTRACT

Language models are ubiquitous in current NLP, and their multilingual capacity has recently attracted considerable attention. However, current analyses have almost exclusively focused on (multilingual variants of) standard benchmarks, and have relied on clean pre-training and task-specific corpora as multilingual signals. In this paper, we introduce XLM-T, a model to train and evaluate multilingual language models in Twitter. In this paper we provide: (1) a new strong multilingual baseline consisting of an XLM-R (Conneau et al. 2020) model pre-trained on millions of tweets in over thirty languages, alongside starter code to subsequently fine-tune on a target task; and (2) a set of unified sentiment analysis Twitter datasets in eight different languages and a XLM-T model fine-tuned on them.

研究动机与目标

• 推动开发一个面向 Twitter 数据的多语言语言模型，以应对领域特定的语言信号（表情符号、俚语等）。
• 提供一个基于 XLM-R 并为 Twitter 做了改造的大规模预训练基线（XLM-Twitter），并发布用于微调和评估的代码。
• 创建一个跨八种语言的统一多语言情感分析基准（UMSAB），以实现公平的跨语言评估。
• 研究零-shot 与数据增强的跨语言迁移，以理解在何种情况下多语言数据比单语言数据更有帮助。

提出的方法

• 通过在 198M 条推文（12B 字符/标记）上继续对 XLM-R 进行预训练，且不对 URL 进行过滤，使用掩码语言模型目标，直到验证收敛（约 14 天，使用 8 张 GPU）。
• 使用适配器对 LM 进行微调（冻结基础 LM，训练一个附加的分类层），以实现高效的多语言情感分类。
• 提供用于推文嵌入提取、微调、推理和评估的起步 Python 代码，集成在 HuggingFace 生态系统中。
• 在八种语言中统一整理并编排 Unified Multilingual Sentiment Analysis Benchmark (UMSAB)，采用平衡的定大小分割（每语言训练 3,033，测试 870 条）。
• 在单语言、零样本跨语言和多语言迁移设置下进行评估，比较 XLM-R 与 XLM-Twitter 在不同任务和语言上的表现。

实验结果

研究问题

• RQ1 How does a Twitter-focused multilingual LM compare to standard multilingual LMs on sentiment analysis tasks across multiple languages?
• RQ2 What is the impact of domain-specific pretraining (Twitter) on multilingual sentiment analysis performance in zero-shot and multilingual transfer settings?
• RQ3 Does a balanced, unified multilingual sentiment benchmark (UMSAB) reveal consistent cross-lingual transfer patterns for Twitter data?
• RQ4 Can adapters enable efficient fine-tuning of a large multilingual LM for Twitter-specific tasks without full model updates?
• RQ5 Which training data strategies (monolingual, bilingual, multilingual) best support cross-lingual sentiment analysis performance?

主要发现

• XLM-Twitter generally outperforms non-Twitter multilingual baselines on multilingual sentiment benchmarks and shows robustness in zero-shot cross-lingual settings.
• In zero-shot experiments, XLM-Twitter achieves strong results across most languages, with notable gains (e.g., Hindi) over XLM-R.
• Cross-lingual transfer with target-language data (monolingual, bilingual, multilingual) shows that including data from multiple languages often helps, and that a single multilingual model offers practicality despite sometimes trading off peak monolingual performance.
• A domain-specific Twitter pretraining signal yields benefits over general-domain multilingual models for social media downstream tasks.
• Emoji and other Twitter-specific signals contribute significantly to semantic representations in tweet embeddings.
• The provided framework and data (UMSAB, XLM-Twitter) facilitate reproducible, multilingual Twitter NLP research and comparisons.