[Paper Review] COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter

Martín Müller, Marcel Salathé+1 more|arXiv (Cornell University)|May 15, 2020

Misinformation and Its ImpactsSocial Sciences135 citations

TL;DR

COVID-Twitter-BERT (CT-BERT) is a domain-specific BERT model pretrained on COVID-19 Twitter data and shows improvements over BERT-Large on five Twitter-based classification tasks, especially for COVID-19 content.

ABSTRACT

In this work, we release COVID-Twitter-BERT (CT-BERT), a transformer-based model, pretrained on a large corpus of Twitter messages on the topic of COVID-19. Our model shows a 10-30% marginal improvement compared to its base model, BERT-Large, on five different classification datasets. The largest improvements are on the target domain. Pretrained transformer models, such as CT-BERT, are trained on a specific target domain and can be used for a wide variety of natural language processing tasks, including classification, question-answering and chatbots. CT-BERT is optimised to be used on COVID-19 content, in particular social media posts from Twitter.

Motivation & Objective

• Motivate improved NLP analysis of COVID-19 content on social media.
• Develop a domain-specific transformer model by pretraining on COVID-19 Twitter data.
• Evaluate CT-BERT against a general-domain model on multiple Twitter-based tasks.

Proposed method

• Pretrain a BERT-Large model on 160 million COVID-19 tweets collected via Crowdbreaks.
• Clean data by removing retweets and duplicates and pseudonymising usernames and URLs.
• Tokenize tweets with a 30,000-word vocabulary and cap sequence length to 96.
• Train with a constant learning rate of 2e-5 and use 10x dupe factor to create 285M training examples.
• Evaluate CT-BERT against BERT-Large on five downstream classification datasets with multiple finetuning runs.
• Provide access to code and model via GitHub for reproducibility.

Experimental results

Research questions

• RQ1 Does domain-specific pretraining on COVID-19 Twitter data improve downstream classification performance compared to a general-domain model?
• RQ2 How large are the gains on COVID-19 related tasks relative to non-COVID Twitter tasks?
• RQ3 What is the relationship between pretraining steps and downstream performance across tasks?

Key findings

• CT-BERT improves mean F1 from 0.802 (BERT-Large) to 0.833 across five datasets.
• Largest relative improvement (ΔMP) is on the COVID-19 Category dataset at 25.88%.
• Vaccine sentiment (MVS) shows a 17.07% ΔMP, SST-2 10.67%, SE 8.97%, and the average ΔMP is 17.57%.
• CT-BERT achieves its strongest gains on health-related, COVID-19 content, particularly on COVID-19 and vaccine-related Twitter data.
• Intermediary pretraining checkpoints show most downstream gains occur by 100–200k steps, with diminishing returns thereafter.