[Paper Review] Covid-Transformer: Detecting COVID-19 Trending Topics on Twitter Using Universal Sentence Encoder

Meysam Asgari-Chenaghlu, Narjes Nikzad-Khasmakhi+1 more|arXiv (Cornell University)|Sep 8, 2020

Advanced Text Analysis Techniques19 references19 citations

TL;DR

This paper proposes a novel, unsupervised framework for detecting trending COVID-19 topics on Twitter by leveraging Universal Sentence Encoder (USE) embeddings for semantic representation, followed by K-means clustering and deep learning-based text summarization. The method outperforms TF-IDF and LDA baselines, achieving 62% F1@10, demonstrating superior semantic clustering and topic coherence through contextual sentence embeddings.

ABSTRACT

The novel corona-virus disease (also known as COVID-19) has led to a pandemic, impacting more than 200 countries across the globe. With its global impact, COVID-19 has become a major concern of people almost everywhere, and therefore there are a large number of tweets coming out from every corner of the world, about COVID-19 related topics. In this work, we try to analyze the tweets and detect the trending topics and major concerns of people on Twitter, which can enable us to better understand the situation, and devise better planning. More specifically we propose a model based on the universal sentence encoder to detect the main topics of Tweets in recent months. We used universal sentence encoder in order to derive the semantic representation and the similarity of tweets. We then used the sentence similarity and their embeddings, and feed them to K-means clustering algorithm to group similar tweets (in semantic sense). After that, the cluster summary is obtained using a text summarization algorithm based on deep learning, which can uncover the underlying topics of each cluster. Through experimental results, we show that our model can detect very informative topics, by processing a large number of tweets on sentence level (which can preserve the overall meaning of the tweets). Since this framework has no restriction on specific data distribution, it can be used to detect trending topics from any other social media and any other context rather than COVID-19. Experimental results show superiority of our proposed approach to other baselines, including TF-IDF, and latent Dirichlet allocation (LDA).

Motivation & Objective

• To detect trending COVID-19-related topics on Twitter in an unsupervised manner using state-of-the-art NLP techniques.
• To address the limitations of traditional topic modeling methods like TF-IDF and LDA, which rely on word-level statistics and fail to capture semantic meaning.
• To develop a framework that leverages deep contextual sentence embeddings to group semantically similar tweets into coherent clusters.
• To generate concise, human-readable summaries of each topic cluster using deep learning-based text summarization.
• To evaluate the model’s performance against established baselines in terms of topic coherence, precision, recall, and F1-score.

Proposed method

• Use the Universal Sentence Encoder (USE) to generate dense, semantic sentence embeddings from individual tweets, preserving contextual meaning.
• Apply K-means clustering on the sentence embeddings to group semantically similar tweets into distinct clusters.
• Employ a deep learning-based text summarization model to generate concise, representative summaries for each cluster.
• Use cosine similarity between sentence embeddings to measure semantic relatedness, enabling meaningful cluster formation.
• Feed the cluster summaries into a topic detection pipeline to extract high-level, interpretable topics from the data.
• Evaluate the framework using precision, recall, and F1-score at top-10 rank, comparing against TF-IDF and LDA baselines.

Experimental results

Research questions

• RQ1 Can a sentence embedding-based approach detect more coherent and semantically meaningful topics than traditional bag-of-words methods like TF-IDF in social media data?
• RQ2 How does the use of Universal Sentence Encoder improve topic clustering performance compared to LDA, which relies on probabilistic word co-occurrence?
• RQ3 To what extent can deep learning-based text summarization enhance interpretability of topic clusters derived from real-time Twitter data?
• RQ4 Does the proposed framework maintain robustness and generalizability beyond the COVID-19 context, given its architecture?
• RQ5 How do the detected topics and keywords compare to ground-truth public concerns during the early pandemic phase?

Key findings

• The proposed model achieved a F1@10 score of 0.62, significantly outperforming TF-IDF (0.35) and LDA (0.51), demonstrating superior topic detection performance.
• The model improved F1-score by over 11% compared to LDA and 27% compared to TF-IDF, indicating stronger semantic clustering and topic coherence.
• Cluster visualization revealed well-separated, meaningful groups with clear semantic centers, unlike the noisy, overlapping clusters produced by TF-IDF.
• The use of semantic embeddings enabled the detection of overlapping topics and nuanced concerns, such as healthcare worker appreciation and misinformation, that word-based methods missed.
• Text summarization of clusters produced concise, interpretable summaries that accurately reflected the core concerns of each topic group.
• The framework demonstrated generalizability beyond COVID-19, as it is not restricted to any specific data distribution or domain.