[Paper Review] BERTuit: Understanding Spanish language in Twitter through a native transformer

Javier Huertas‐Tato, Alejandro Martín+1 more|arXiv (Cornell University)|Apr 7, 2022

Misinformation and Its Impacts4 citations

TL;DR

BERTuit is a Spanish-specific BERT-based transformer model pre-trained on 230 million tweets using RoBERTa optimization, enabling superior performance on informal Spanish text in Twitter. It outperforms multilingual models like XLM-RoBERTa and M-BERT in tasks like misinformation detection, irony recognition, and author profiling, demonstrating strong zero-shot capability and effectiveness even with limited data.

ABSTRACT

The appearance of complex attention-based language models such as BERT, Roberta or GPT-3 has allowed to address highly complex tasks in a plethora of scenarios. However, when applied to specific domains, these models encounter considerable difficulties. This is the case of Social Networks such as Twitter, an ever-changing stream of information written with informal and complex language, where each message requires careful evaluation to be understood even by humans given the important role that context plays. Addressing tasks in this domain through Natural Language Processing involves severe challenges. When powerful state-of-the-art multilingual language models are applied to this scenario, language specific nuances use to get lost in translation. To face these challenges we present extbf{BERTuit}, the larger transformer proposed so far for Spanish language, pre-trained on a massive dataset of 230M Spanish tweets using RoBERTa optimization. Our motivation is to provide a powerful resource to better understand Spanish Twitter and to be used on applications focused on this social network, with special emphasis on solutions devoted to tackle the spreading of misinformation in this platform. BERTuit is evaluated on several tasks and compared against M-BERT, XLM-RoBERTa and XLM-T, very competitive multilingual transformers. The utility of our approach is shown with applications, in this case: a zero-shot methodology to visualize groups of hoaxes and profiling authors spreading disinformation. Misinformation spreads wildly on platforms such as Twitter in languages other than English, meaning performance of transformers may suffer when transferred outside English speaking communities.

Motivation & Objective

• To address the limitations of general-purpose multilingual language models in understanding informal, context-rich Spanish Twitter text.
• To develop a specialized transformer model trained exclusively on massive volumes of native Spanish tweets to capture linguistic nuances, irony, and cultural context.
• To enhance the detection and analysis of misinformation on Twitter by leveraging fine-grained contextual embeddings from a domain-optimized model.
• To enable effective zero-shot visualization of hoax clusters and profiling of disinformation spreaders using semantic embeddings.

Proposed method

• Pre-training BERTuit from scratch using the BERT-base architecture on a curated dataset of 230 million Spanish tweets from the Twitter Archive.
• Applying RoBERTa-style optimization techniques, including dynamic masking and next-sentence prediction, to improve contextual representation learning.
• Using masked language modeling as the pre-training objective to capture syntactic and semantic patterns specific to informal Spanish on Twitter.
• Generating contextualized tweet embeddings via the final transformer layer for downstream task representation.
• Applying t-SNE for dimensionality reduction to project high-dimensional embeddings into a 2D space for visual analysis of misinformation clusters.
• Averaging tweet-level embeddings per user to create user-level representations for author profiling and classification.

Experimental results

Research questions

• RQ1 Can a monolingual, Twitter-optimized transformer model outperform multilingual models in understanding informal Spanish on social media?
• RQ2 How effective is BERTuit in detecting irony, hate speech, and sentiment in Spanish Twitter text compared to state-of-the-art multilingual models?
• RQ3 To what extent can BERTuit enable zero-shot visualization of misinformation clusters using embedding-based clustering and dimensionality reduction?
• RQ4 How much training data is required per user to achieve high accuracy in classifying disinformation spreaders using BERTuit embeddings?
• RQ5 What role does data diversity (number of users) play compared to data quantity (number of tweets per user) in training effective classification models?

Key findings

• BERTuit outperformed XLM-RoBERTa, M-BERT, and XLM-T across multiple NLP tasks on Spanish Twitter, including hate speech detection, irony detection, and sentiment analysis.
• With just one tweet per user, the model achieved over 74% accuracy in classifying disinformation spreaders, demonstrating strong few-shot learning capability.
• The number of distinct users in the training set had a stronger positive correlation with model accuracy than the number of tweets per user, highlighting the importance of data diversity.
• t-SNE visualization of BERTuit embeddings revealed meaningful clustering of hoax-related content, enabling effective zero-shot identification of misinformation groups.
• The model successfully captured stylistic and semantic patterns from minimal data, enabling robust author profiling even with sparse tweet samples.
• The results confirm that domain-specific pre-training on social media data significantly improves performance on informal language tasks compared to general multilingual models.